SPPD : Self-training with Process Preference Learning Using Dynamic Value Margin (2025)

Abstract: Enhancing the numerical and logical reasoning capabilities of Large Language Models (LLMs) has become a prominent research focus. Existing approaches exhibit notable limitations: inference-phase techniques, such as Chain of Thought, depend on prompt engineering and pretrained knowledge; sentence-level Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) struggle to ensure step-wise mathematical correctness and often rely on model distillation or human annotations; Reinforcement Learning (RL) methods entail high GPU memory consumption and training instability. To overcome these challenges, we propose S elf-training with P rocess P reference learning using D ynamic value margin ( SPPD ). SPPD formulates reasoning as a process-based Markov Decision Process (MDP), leveraging the Bellman optimality equation to derive a dynamic value margin for step-level preference optimization. It further incorporates tree-based self-sampling of model responses, eliminating the need for distillation. We theoretically establish that SPPD is equivalent to on-policy policy gradient methods under constrained reward functions. Experimental results on 7B-scale models show consistent superiority across both in-domain and out-of-domain mathematical benchmarks.

## introduction recent advancements in o-series models @xcite have significantly enhanced the mathematical reasoning capabilities of large language models (llms), positioning numerical and logical reasoning as a focal point in current research @xcite @xcite @xcite @xcite . to enhance model reasoning during the inference phase, techniques such as chain of thoughts * corresponding author. to address these challenges, we introduce selftraining with process preference learning using dynamic value margin (sppd). unlike sft and dpo, sppd optimizes preferences at the step level via dynamic value margins, eliminating the need for distillation. specifically, sppd utilizes a process-based markov decision process (mdp) and the bradley-terry (bt) @xcite , employing the bellman optimality equation @xcite ) and an online rl objective modeled with mdp @xcite . this approach enables step-wise dpo without relying on stronger models for data distillation. instead, it employs tree search methods that sample trajectories solely from the model's responses and logits scores. additionally, a strategy combining sft and dpo based on process reward model(prm) rejection sampling enhances training efficiency, progressively refining reasoning capabilities. theoretically, our method is equivalent to an on-policy policy gradient method under specific reward constraints. experimental results demonstrate that sppd achieves great improvements across various architectures and datasets, outperforming many opensource and some closed-source models. our contributions include: ## preliminaries in this section, we first define the step-level mdp in natural language processing. subsequently, based on this step-level mdp, we reformulate the original reinforcement learning from human feedback (rlhf) objective and derive its optimal (fixedpoint) solution to the maximum causal entropy problem. ## method in this section, we first introduce a tree-based selfsampling method to generate step trajectories sharing common prefixes without distillation. we further incorporate sentence-level sft and dpo using the prm, aiming to enhance the smoothness and effectiveness of model training. finally, step preference learning with dynamic value margins is proposed and further refined based on reward equivalence. ## theoretical analysis in this section, we establish the equivalence between offline step dpo and online policy gradient under a specific reward formulation. lemma 5.1 (online policy gradient on π p θ (see definition c.1); @xcite ). for any mdp, the expected long-term reward under π p θ is given by j(θ) = τ π p θ (τ )r(τ ), where r(τ ) denotes the cumulative reward along trajectory τ . the corresponding policy gradient is: theorem 5.2 (equivalence between offline step dpo and online policy gradient). define the trajectory reward in equation (7) as: r(τ ) = t+1 @xmath0 let the offline every-step preference loss be defined as: then, the following identity holds: the proof is provided in appendix e.3. ## conclusion in this work, we propose sppd, a self-training with process preference learning using dynamic value margin. sppd utilizes the bellman optimality equation and the online rl objective modeled with mdp and designs a step-level tree self-sampling scheme without any distillation. moreover, we propose a sft and dpo scheme using prm for rejection sampling, making the training of sppd smothor and more effective. finally, we theoretically demonstrate that under specific reward constraints, our method is equivalent to on-policy policy gradient optimization.