The Lessons of Developing Process Reward Models in Mathematical Reasoning (2025)

Abstract: Process Reward Models (PRMs) aim to identify and mitigate intermediate errors in the reasoning processes in mathematical reasoning of Large Language Models (LLMs).However, the development of effective PRMs faces significant challenges, particularly in data annotation and evaluation methodologies.In this paper, through extensive experiments, we demonstrate that commonly used Monte Carlo (MC) estimation-based data synthesis for PRMs typically yields inferior performance and generalization compared to LLM-as-a-judge and human annotation methods.Furthermore, we identify potential biases in conventional Best-of-N (BoN) evaluation strategies for PRMs.To address these challenges, we develop a consensus filtering mechanism that effectively integrates MC estimation with LLM-as-a-judge and advocates a more comprehensive evaluation framework that combines response-level and step-level metrics. Based on the mechanisms, we significantly improve both model performance and data efficiency in the BoN evaluation and the step-wise error identification task.Finally, we release a new state-of-the-art PRM that outperforms existing open-source alternatives and provides practical guidelines for future research.

## introduction in recent years, large language models (llms) have made remarkable advances in mathematical reasoning @xcite @xcite @xcite ), yet they can make mistakes, leading to wrong conclusions. moreover, even when achieving correct final answers, these powerful models can still regularly use flawed reasoning steps, which undermine the reliability and trustworthiness of llms' reasoning processes. to address these challenges, @xcite @xcite ) are proposed to identify and mitigate process errors, thereby enabling finer-grained supervision on the reasoning process. one critical challenge of developing prms lies in the data annotation for the correctness of reasoning processes, which is typically expensive and time-consuming. while @xcite recruited human annotators with detailed instructions and elaborate procedures to achieve satisfactory annotation quality, the prohibitive cost pushes researchers to explore automated annotation methods. among them, one commonly used approach is to assess process correctness by estimating the empirical probability of leading to the correct final answers through monte carlo (mc) methods, which has attracted great research interests and has also been commonly employed in practice @xcite @xcite . another challenge lies in evaluating prm performance, as previous studies @xcite @xcite have predominantly relied on the best-of-n (bon) evaluation, which selects the highest-scored response from n candidates according to a prm. recently, pro-cessbench @xcite have emerged to evaluate the capability of prms in identifying step-wise correctness. nevertheless, during the training of our own prm following conventional principles to construct data using mc estimation and evaluate on bon, we gain several crucial lessons. in terms of mc estimation, (1) we observe that the prm trained via mc estimation demonstrated significantly inferior performance and generalization capabilities compared to llm-as-a-judge @xcite and human annotation. (2) we attribute the suboptimal performance of mc estimation to its fundamental limitation, which attempts to evaluate deterministic current-step correctness based on potential future outcomes. it significantly relies on the performance of the completion model, m a t h -s h e p h e r d -p r m -7 b r l h f l o w -p r m -m i s t r a l -8 b r l h f l o w -p r m -d e e p s e e k -8 b s k y w o r k -p r m -1 . 5 b s k y w o r k -p r m -7 b e u r u s p r m -s t a g e 1 e u r u s p r m -s t a g e 2 q w e n 2 . 5 -m a t h -7 b -m a t h -s h e p h e r d q w e n 2 . 5 -m a t h -7 b -p r m 8 0 0 k q w e n 2 . 5 -m a t h -p r m -7 b q w e n 2 . 5 -m a t h -p r m -7 2 b figure 1: overview of evaluation results on the best-of-8 strategy of the policy model qwen2.5-math-7b-instruct and the benchmark processbench @xcite across multiple prms (see regarding the bon evaluation, (1) the unreliable policy models generate responses with correct answers but flawed processes, leading to a misalignment between the outcome evaluation criteria of bon and the prm objectives of process verification. to address these challenges, we propose a consensus filtering mechanism that combines mc estimation with llm-as-a-judge, retaining only instances where both agree on error locations in the solution. our approach improves both data efficiency and performance over existing prms in the conventional bon evaluation. furthermore, we advocate for complementing response-level bon with step-wise evaluation methods. we employ the step-wise benchmark processbench @xcite to measure the ability to identify process errors. our trained prms exhibit impressively stronger error identification performance than other open-source models, from prms to general language models, confirming that our training ap-proach genuinely teaches prms to assess the correctness of intermediate reasoning steps. ## preliminary trials in this section, we describe our preliminary attempts to train prms via mc estimation-based reasoning step annotation. despite our efforts in scaling up training data and careful tuning of training objectives, we found that the mc estimationbased prms do not possess noticeable advantages over the one trained on human-annotated data, and even lag significantly behind the latter in identifying specific erroneous reasoning steps. ## the lessons in this section, we present the critical lessons gained during the prm training comprising two main aspects: (1) the limitations of commonly adopted mc estimation approaches in prms training, and (2) the bias in using bon as the sole evaluation metric for optimizing prms. ## our prms this section presents our methodology for overcoming the previously discussed limitations and the details of our trained prm achieving state-ofthe-art performance. ## conclusion in this paper, we present the critical lessons gained during developing prms and release a new stateof-the-art prm. firstly, we identify critical limitations in current data construction approaches for prms, demonstrating that mc estimation-based data construction yields inferior performance and generalization compared to llm-as-a-judge and human annotation. then we reveal the potential bias in using response-level bon evaluation alone for prms and advocate for combining both response-level and step-level metrics. to address these issues, we propose an effective consensus filtering strategy combining mc estimation with llm-as-a-judge. our evaluation, incorporating both response-level bon and identifying step-wise correctness task processbench, demonstrates significant improvements in data efficiency and model performance. limitation there are several limitations remained in our current work. firstly, there exists a considerable performance gap between our prms and the bon upper bound (pass@8), suggesting substantial optimization potential. then the best practices for utilizing prms in reinforcement learning remain unexplored. finally, although our approach combines llm-as-a-judge with mc estimation for consensus filtering, the efficient utilization of existing high-quality human annotation data is still largely under-explored. for instance, gradually expanding high-quality datasets through weakly supervised methods can be investigated as a promising direction for future exploration.