Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision (2025)

Abstract: Large Language Models (LLMs) are prone to hallucination, especially during multi‐hop and reasoning-intensive tasks such as mathematical problem solving. While Outcome Reward Models verify only final answers, Process Reward Models (PRMs) score each intermediate step to steer generation toward coherent solutions. We introduce PathFinder‐PRM, a novel hierarchical, error‐aware discriminative PRM that first classifies math and consistency errors at each step, then combines these fine‐grained signals to estimate step correctness. To train PathFinder‐PRM, we construct a 400K‐sample dataset by enriching the human‐annotated PRM800K corpus and RLHFlow Mistral traces with three‐dimensional step‐level labels. On PRMBench, PathFinder‐PRM achieves a new state‐of‐the‐art PRMScore of 67.7, outperforming the prior best (65.5) while using 3× less data. When applied to reward guided greedy search, our model yields prm@8 48.3, a +1.5 point gain over the strongest baseline. These results demonstrate that decoupled error detection and reward estimation not only boost fine‐grained error detection but also substantially improve end‐to‐end, reward‐guided mathematical reasoning with greater data efficiency. Our code is available at https://github.com/declare-lab/PathFinder-PRM.

## introduction large language models (llms) have achieved remarkable success on many natural language tasks, including open-ended generation and complex reasoning @xcite . however, they remain prone to hallucinations and subtle logical errors when generating multi-step solutions, particularly in domains such as mathematical problem solving @xcite . traditional outcome-only verifiers (outcome reward models) can check a final answer but fail to catch intermediate missteps that lead reasoning astray @xcite . to address this gap, process reward models (prms) have been proposed, which assign individual rewards to each reasoning step. @xcite . as such, prms can filter out erroneous chains of thought and guide generation toward more reliable reasoning trajectories @xcite . recent interest in explicitly reasoning-centric llms such as deepseek-r1 and openai's gpt-o series models underscores the field's growing emphasis on human-like thinking and the ability to flexibly scale test-time compute @xcite . these models demonstrate extended deliberation and use structured reasoning traces to solve complex problems. in such settings, effective process supervision is crucial: rather than merely verifying a final answer, it must guide and correct the reasoning process at every step, ensuring logical coherence and factual accuracy throughout. prms are thus an essential component in aligning reasoning llms with reliable multi-step reasoning. despite recent advances, current prms still struggle with fine-grained error types. for example, the prmbench benchmark reveals that many state-of-the-art prms fall short of detecting subtler faults such as non-redundancy violations, domain inconsistencies, or deceptive logical steps @xcite . moreover, existing methods typically combine error detection (is this step wrong?) with path optimality (how helpful is this step in reaching the solution?) in a single prediction, leaving each signal underutilized @xcite . in this work, we argue that error detection and value estimation are complementary but distinct objectives. by decoupling them into two sequential subtasks, first explicitly identifying specific error categories and using those error signals to compute a step-level reward, we can obtain richer supervision and stronger guidance for downstream generation (depicted in figure 1 ). to this end, we introduce pathfinder-prm, an error-aware hierarchical prm that (1) classifies each step to detect the presence of math or consistency errors, and (2) combines these fine-grained error labels to produce a final reward score. we construct a new training corpus by augmenting the human-annotated prm800k @xcite and automated rlhflow mistral data @xcite with our three-dimensional labels, yielding around 400k richly annotated reasoning trajectories. our experiments on processbench @xcite , prmbench @xcite , and a suite of end-to-end math benchmarks show that pathfinder-prm not only establishes the new state of the art among prms trained on prm800k-only data but also continues to scale gracefully when incorporating additional automated annotations. we summarize our main contributions: generative prms scale verifier compute, utilizing the language modeling head of the prms to generate a chain of thought (cot) before producing step-level correctness classifications. associated training data is typically obtained by filtering llm-as-a-judge reasoning traces against human step-level labels @xcite or relative progress estimation @xcite . limitations of current prms: current prms face significant challenges in detecting nuanced error types. although frontier models excel at identifying obvious mistakes @xcite , their performance deteriorates markedly when confronted with more subtle error types. the recently introduced prmbench benchmark demonstrates this limitation, revealing substantial performance drops across fine-grained error categories such as redundancy, circular logic, step inconsistency and domain inconsistency @xcite . ## conclusion in this work, we introduced pathfinder-prm, a hierarchical discriminative process reward model that decouples error detection from step optimality guidance, classifying math and consistency errors before computing step rewards. our evaluation and ablation trials demonstrate that error-aware hierarchical supervision yields notable improvements on prm benchmarks, with pathfinder-prm-7b achieving state-of-the-art performance among discriminative prms on prmbench and strong results on processbench despite using less than three times the data used to train the current best performing model. a similar performance boost is observed in reward-guided search evaluation, and we further show that these gains scale consistently when paired with larger policy models, affirming our hypothesis about the efficacy of hierarchical error-aware reward generation. our approach is, therefore, a promising direction for more robust and interpretable process reward models, with potential for further gains when scaled to larger architectures.