C ode PRM : Execution Feedback-enhanced Process Reward Model for Code Generation (2025)

Abstract: Code generation is a critical reasoning task for large language models (LLMs). Recent advancements have focused on optimizing the thought process of code generation, achieving significant improvements. However, such thought process lacks effective process supervision, making it hard to optimize the thoughts. Although Process Reward Models (PRMs) have been widely established in mathematical reasoning, building a code PRM is still not trivial for the gap between thoughts to code. In this paper, we propose CodePRM, a novel approach that leverages the code execution feedback to build a code PRM. Specifically, we first collect a large dataset of thought traces, where each thought step is labeled with their derived code’ pass rates, accompanied by the corresponding code snippets, and execution feedback. During training, we train a PRM to take both the reasoning process and code execution feedback as input to score individual thought steps, enabling it to leverage code execution results to distinguish between high-quality and low-quality thought steps. Finally, to use the PRM during inference, we develop a Generate-Verify-Refine (GVR) pipeline where the CodePRM serves as a process verifier to dynamically identify and correct errors in the thought process during code search. Experimental results demonstrate that CodePRM with the inference algorithm outperforms strong baselines, significantly enhancing code generation performance. Further analysis reveals the key factors for building a code PRM.

## introduction code generation is a task that demands advanced reasoning capabilities @xcite @xcite . early efforts to enhance llms' performance in code generation primarily focused on direct optimizations at the code level @xcite , such as decomposing code @xcite or modifying code based on variable values change @xcite . recently, however, there has been a shift in focus toward improving the underlying thought processes that drive code generation. numerous tree search algorithms have been proposed to explore and refine these thought processes @xcite @xcite , demonstrating significant potential and achieving promising results. despite these advancements, a significant gap remains in developing effective process supervision for the thought processes involved in code generation. unlike mathematical reasoning, where process reward models (prms) have been extensively studied, research on prms for code generation is limited. this gap stems from two key challenges: (1) defining intermediate steps: in code generation, the granularity of a "step" lacks a consistent definition. a step could be an element in the code space (e.g., a line of code @xcite or a token of code @xcite ) or a natural language-based reasoning step @xcite . recent studies suggest that focusing on thought steps is more effective for llms in code generation @xcite . (2) mapping code errors to thought processes: in mathematics, intermediate steps often contain partial computation results, making it easier to trace errors in the final answer back to middle thoughts. while in code generation, the thoughts are usually abstract and involve algorithmic design, workflow construction, and data structure definition, making it harder to pinpoint errors in the thought process that lead to faulty code. empirically, as illustrated in figure 1 , we find that training a code prm using the same paradigm as mathematical prms performs no better than random reward assignment, demonstrating that developing an effective code prm requires innovative paradigms tailored to the unique challenges of code generation. this paper presents a new perspective on the problem. the idea is that thought processes in code generation should not be evaluated in isolation; instead, they should be assessed in conjunction with the generated code and execution feedback. the feedback in the code environment usually contains detailed error information about the code, which would be helpful in identifying process errors. in light of this, we propose leveraging the code and execution feedback as auxiliary information to construct a robust codeprm. specifically, we first employ tree search algorithms to generate and collect a dataset of labeled thought steps for code generation. each thought step is annotated with the average pass rates of the code derived from it. during this process, we also store the corresponding code snippets and execution feedback associated with each thought. subsequently, we train a vanilla llm into a feedback-enhanced codeprm through supervised fine-tuning. codeprm incorporates both the code and execution feedback as auxiliary inputs when scoring thought steps. as illustrated in figure 1 , this new paradigm delivers significant performance improvements. to apply codeprm in inference, we design a generate-verify-refine (gvr) framework. in this framework, a generator llm explores diverse thought steps and corresponding code using tree search algorithms. codeprm then acts as a verifier to identify erroneous thought steps. the erroneous thoughts would be identified and then refined by the generator llm, which ultimately leads to improved code. overall, our key contributions are as follows: ## method the workflow of our method is illustrated in figure 2. first, we collect intermediate thought steps data for code, where each thought is annotated with the average pass rate of the code it subsequently derives. with the code and execution feedback as an auxiliary input, we fine-tune an llm through supervised fine-tuning to develop a feedback-enhanced prm. during inference, we propose a generate-verify-refine (gvr) pipeline, which can be integrated with various search algorithms. in this pipeline,codeprm serves as a thought verifier, identifying incorrect thoughts during the search. these erroneous thoughts are then refined to produce higher-quality code. ## experiments in this section, we conduct experiments to illustrate the effectiveness of codeprm compared with previous methods and investigate the properties of building and using a code prm.