Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning (2025)

Abstract: Many studies focus on data annotation techniques for training effective PRMs. However, current methods encounter a significant issue when applied to long CoT reasoning processes: they tend to focus solely on the first incorrect step and all preceding steps, assuming that all subsequent steps are incorrect. These methods overlook the unique self-correction and reflection mechanisms inherent in long CoT, where correct reasoning steps may still occur after initial reasoning mistakes. To address this issue, we propose a novel data annotation method for PRMs specifically designed to score the long CoT reasoning process. Given that under the reflection pattern, correct and incorrect steps often alternate, we introduce the concepts of Error Propagation and Error Cessation, enhancing PRMs’ ability to identify both effective self-correction behaviors and reasoning based on erroneous steps. Leveraging an LLM-based judger for annotation, we collect 1.7 million data samples to train a 7B PRM and evaluate it at both solution and step levels. Experimental results demonstrate that compared to existing open-source PRMs and PRMs trained on open-source datasets, our PRM achieves superior performance across various metrics, including search guidance, BoN, and F1 scores. Compared to widely used MC-based annotation methods, our annotation approach not only achieves higher data efficiency but also delivers superior performance. Detailed analysis is also conducted to demonstrate the stability and generalizability of our method.

## introduction mathematical reasoning has become a crucial metric for evaluating the intelligence of llms @xcite @xcite , garnering substantial attention from researchers in recent years. while numerous studies have focused on enhancing llms' mathematical reasoning capabilities @xcite , llms continue to exhibit limitations in practical applications, including calculation errors, flawed derivations, and logical errors. prms help address these challenges by providing fine-grained evaluation signals for the intermediate steps in the llm reasoning process, indicating the correctness of each step. in the short chain-of-thought (cot) pattern, llms lack self-reflection abilities. this means that once a mistake is made, all following steps are likely to be wrong. the prm only needed to identify correct-to-correct and correct-to-incorrect transitions. therefore, conventional prm data construction methods @xcite ) use all steps from correct solutions, but only use steps up to the first error step in incorrect solutions. however, as shown in appendix d, long cot llms often make mistakes during reasoning, but they can later correct themselves or come up with alternative solutions. therefore, capturing this incorrect-to-correct pattern is crucial for prms to better evaluate the intermediate steps in long cot reasoning. currently, the monte carlo (mc) method @xcite is widely used for automatically labeling intermediate steps. however, when dealing with long cot patterns, the mc method has the following drawbacks: (1) inaccurate labeling of intermediate reasoning steps. mc method rolls out multiple paths from an intermediate step. if one path reaches the correct answer, it assumes the step is correct. however, long cot reasoning processes often involve self-correction or reflection, so the final answer doesn't always reflect the correctness of intermediate steps. (2) high computational cost. the mc method requires multiple rollouts for each intermediate step. in the long cot pattern, numerous intermediate steps and lengthy rollouts can greatly increase the computational burden. to address the shortcomings of current prm de-signs and mc methods in long cot scenarios, we propose a new data annotation approach. first, we introduce two fundamental rules specifically for long cot: error propagation and error cessation. these rules capture two key patterns: reasoning based on incorrect steps and self-correction after making mistakes. we then incorporate these rules into the llm judge's prompt to guide the model in annotating intermediate steps of the reasoning process. finally, we use the annotated data to train the prm. experimental results demonstrate that compared to existing open-source prms and prms trained on open-source datasets, our prm achieves the best performance at both the solution level and the step level. at the solution level, we use the traditional best-of-n (bon) method to see how well prms can select the best answer from several options. since bon doesn't make full use of the process reward signals that prms provide, we also propose a new metric that measures whether incorporating these signals into step-level search increases the probability of discovering the correct solution. at the step level, we establish a test set through the cross-validation of two distinct annotation methods (o1 model @xcite and manual annotation), enabling rigorous assessment of prms in evaluating the correctness of individual solution steps. across both solution-level and step-level evaluation metrics on the math500 and aime24, our prm consistently outperforms all baselines. furthermore, we conduct a comprehensive comparison with the mc-based data annotation method, demonstrating that our method not only achieves greater data efficiency but also delivers superior performance. an in-depth analysis of our method is also conducted to further demonstrate its stability and generalizability. in summary, our contributions are as follows: long chain-of-thought reasoning language models llms have demonstrated remarkable reasoning capabilities for complex tasks. one pivotal method to improve the reasoning ability of llms is chain-of-thought (cot) @xcite @xcite , which significantly improves performance by guiding llms to generate intermediate reasoning steps. initial research @xcite @xcite on cot mainly focuses on developing effective prompt engineering techniques. openai o1 (ope-nai, 2024) is the first to introduce inference time scaling law, which employs reinforcement learning to encourage models to generate additional reasoning tokens, thereby overcoming more challenging tasks. in long cot paradigm, llms can decompose problems, explore multiple pathways, and automatically correct reasoning errors. several efforts @xcite @xcite have successfully replicated this powerful reasoning ability. however, prms specifically tailored for this long cot paradigm remain underexplored. ## disadvantages of current prms previous works on training prms primarily focus on the first incorrect step and assume that all subsequent steps are reasoned based on this erroneous step, thus deeming them all incorrect. however, in a long cot pattern, self-correction behaviors often occur, where correct reflective steps can still follow an incorrect step. this results in a distribution shift between the prm training data and the data encountered during inference in long cot scenarios, potentially affecting performance. therefore, we aim to investigate: does the self-correction behavior of long cot models impact the performance of current prms? to answer this question, we construct two test sets: an error-free set (ef set) and a reflection-based set (rb set). the final answers of solutions for both sets are correct. the key difference lies in the correctness of the intermediate reasoning steps. all reasoning steps in the error-free set are correct, while solutions in the reflection-based set contain erroneous intermediate steps and perform self-correction. we evaluate several open-source prms on these two test sets: qwen2.5-math-prm-7b @xcite , mathshepherd-prm-7b @xcite , and skywork-prm-7b (o1 team, 2024), focusing on the accuracy of prms in determining whether the solutions are correct. our experimental results, as presented in ## method in this section, we introduce a framework for constructing prm data tailored to reflective reasoning. first, we introduce how to train a long cot model that can autonomously segment its output based on semantics. then, we describe our detailed step-bystep annotation guidelines and the implementation of prm training. the overall framework of our method is shown in figure 1 . ## experiments in this section, we present the experimental results of our prm in comparison with (1) open-source prms and prms trained on open-source datasets (section5.2), and (2) prms trained on data generated through mc-based methods (section5.3).