Better Process Supervision with Bi-directional Rewarding Signals (2025)

Abstract: Process supervision, i.e., evaluating each step, is critical for complex large language model (LLM) reasoning and test-time searching with increased inference compute. Existing approaches, represented by process reward models (PRMs), primarily focus on rewarding signals up to the current step, exhibiting a one-directional nature and lacking a mechanism to model the distance to the final target. To address this problem, we draw inspiration from the A* algorithm, which states that an effective supervisory signal should simultaneously consider the incurred cost and the estimated cost for reaching the target. Building on this key insight, we introduce BiRM, a novel process supervision model that not only evaluates the correctness of previous steps but also models the probability of future success. We conduct extensive experiments on mathematical reasoning tasks and demonstrate that BiRM provides more precise evaluations of LLM reasoning steps, achieving an improvement of 3.1% on Gaokao2023 over PRM under the Best-of-N sampling method. Besides, in search-based strategies, BiRM provides more comprehensive guidance and outperforms ORM by 5.0% and PRM by 3.8% respectively on MATH-500.

## introduction with the rapid development of llms, how to supervise them has become a key research challenge, especially for complex tasks like long-term reasoning @xcite @xcite . previous work has explored training process supervision models to provide dense supervision on each step @xcite @xcite , which is intuitively and practically better than outcome supervision models insufficient evaluation prm struggles in later stages vm is inaccurate in early stages ## conclusion in this work, we introduce birm, a novel process supervision model for large language models (llms), inspired by the a* algorithm. birm provides bidirectional supervision signals, evaluating both the correctness of reasoning steps taken so far and the probability of reaching correct answers in the future. our extensive experiments demonstrate the effectiveness of birm across various mathematical reasoning tasks, outperforming existing supervision models like orm and prm. through de-tailed analysis, we highlight the strengths of birm in guiding the search process and improving solution re-ranking. we hope that our approach contributes valuable insights to the field of process supervision and opens avenues for future research in enhancing llm-based reasoning.