Full-Step- DPO : Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning (2025)

Abstract: Direct Preference Optimization (DPO) often struggles with long-chain mathematical reasoning. Existing approaches, such as Step-DPO, typically improve this by focusing on the first erroneous step in the reasoning chain. However, they overlook all other steps and rely heavily on humans or GPT-4 to identify erroneous steps. To address these issues, we propose Full-Step-DPO, a novel DPO framework tailored for mathematical reasoning. Instead of optimizing only the first erroneous step, it leverages step-wise rewards from the entire reasoning chain. This is achieved by training a self-supervised process reward model, which automatically scores each step, providing rewards while avoiding reliance on external signals. Furthermore, we introduce a novel step-wise DPO loss, which dynamically updates gradients based on these step-wise rewards. This endows stronger reasoning capabilities to language models. Extensive evaluations on both in-domain and out-of-domain mathematical reasoning benchmarks across various base language models, demonstrate that Full-Step-DPO achieves superior performance compared to state-of-the-art baselines.

## introduction large language models (llms) have attracted massive interest due to their remarkable capabilities across various tasks @xcite @xcite @xcite @xcite . however, they commonly encounter difficulties when tackling complex and symbolic multi-step reasoning, particularly in mathematical problem reasoning @xcite @xcite . to improve the mathematical reasoning ability, some studies use direct preference optimization (dpo) @xcite ) with solution-wise preference data but find its benefit limited @xcite @xcite . recent works attribute this limitation to dpo's inability to perform process supervision and instead builds preference data based on reasoning steps rather than entire solutions @xcite @xcite . for example, step-dpo @xcite focuses on optimizing only the first erroneous step in the reasoning chain, demonstrating notable improvements. however, despite their improvements, these existing methods face the following limitations: as a result, they fail to fully optimize the reasoning chains, leading to suboptimal performance. to address the above limitations, we propose full-step-dpo, a novel dpo framework for mathematical reasoning. as illustrated in figure 1 , unlike vanilla dpo, which operates solution-wise, or step-dpo, which focuses solely on the first erroneous step, full-step-dpo utilizes each step in the entire reasoning chain and optimizes them using step-wise rewards. we first train a process reward model (prm) @xcite in a self-supervised way, utilizing data generated by the model itself. this approach enables the prm to automatically score each step in the reasoning chain, eliminating the reliance on external annotations such as gpt-4 or humans. then, we propose a novel step-wise dpo loss, which employs dynamic gradient updates to optimize each step based on its corresponding reward. this approach shifts the optimization focus from solutionwise to step-wise, enabling the policy model to achieve superior reasoning capabilities. we conduct experiments on both in-domain and out-of-domain mathematical reasoning datasets with four widely used backbone llms. experimental results demonstrate that our full-step dpo consistently outperforms the dpo and step-dpo baselines, validating its effectiveness in enhancing reasoning performance. our contributions can be summarized as follows: • we propose the full-step-dpo framework with a novel step-wise dpo loss that dynamically adjusts each step's gradient based on its reward, enabling step-wise optimization rather than solution-wise and enhancing reasoning ability. preference learning recently, preference learning @xcite has attracted significant attention due to its ability to align with human preferences and distinguish between positive and negative examples. while these methods, like dpo @xcite , have proven effective in general domains, it offers only marginal benefits for mathematical reasoning @xcite . some works @xcite suggest that dpo's focus on coarse solution-wise preferences makes it less effective at correcting errors in multi-step reasoning, hindering reasoning improvement. therefore, step-dpo (lai et al., 2024) was proposed, which first identifies the first erroneous step, and then optimizes only this erroneous step along with the corresponding correct one. although this approach enhances mathematical reasoning capabilities, it totally overlooks the other steps in long-chain reasoning, which also provide valuable information and should not be completely disregarded. building on this consideration, we propose full-step-dpo, which fully accounts for each step by dynamically optimizing all steps in the reasoning process. step-wise supervision recent findings by lightman et al. (2023) suggest that step-wise supervision outperforms outcome-wise, due to the provision of more detailed feedback. however, training a prm requires either costly manual annotation (lightman et al., 2023) or significant computational resources (khalifa et al., 2023; wang et al., 2023c), which hinders the advancement and practical application of prm. therefore, in this paper, we aim to build a prm for mathematical reasoning without relying on human annotation and with reduced computational resources. additionally, we explore the effectiveness of the prm in decoding and preference learning scenarios. 3 full-step dpo in this section, we elaborate the proposed full-step dpo framework. we begin by reviewing the background of previous dpo and step-dpo. then we introduce the novel step-wise dpo loss which optimizes with step-wise rewards, and the process reward model which automatically generate these step-wise rewards. finally we outline the complete training pipeline of our full-step-dpo.