Process-based Self-Rewarding Language Models (2025)

Abstract: Large Language Models have demonstrated outstanding performance across various downstream tasks and have been widely applied in multiple scenarios. Human-annotated preference data is used for training to further improve LLMs’ performance, which is constrained by the upper limit of human performance. Therefore, Self-Rewarding method has been proposed, where LLMs generate training data by rewarding their own outputs. However, the existing self-rewarding paradigm is not effective in mathematical reasoning scenarios and may even lead to a decline in performance. In this work, we propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization within the self-rewarding paradigm. Our new paradigm successfully enhances the performance of LLMs on multiple mathematical reasoning benchmarks through iterative Process-based Self-Rewarding, demonstrating the immense potential of process-based self-rewarding to achieve LLM reasoning that may surpass human capabilities.

## introduction large language models (llms) acquire powerful multi-task language capabilities through pretraining on extensive corpus @xcite . additionally, supervised finetuning (sft) can further effectively improve the model's performance on end-tasks. however, it is found that models after sft are prone to hallucinations @xcite due to the simultaneous increasing of the probabilities of both preferred and undesirable outputs @xcite . therefore, to further enhance the language capabilities of llms to align with human-level performance effectively, researchers often utilize human-annotated preference data for training. a representative approach is reinforcement learning from human feedback (rlhf) @xcite , which utilizes rl algorithms and external reward signals to help llms learn specific preferences. however, most reward signals rely on human annotations or reward models, which is expensive and bottlenecked by human capability and reward model quality. so the self-rewarding language models paradigm @xcite is proposed to address situations with insufficient labeled data, which integrates the reward model and the policy model within the same model. in this framework, a single model possesses the ability to both perform the target task and provide reward feedback. the model can execute different tasks based on the scenario and conduct iterative updates. this paradigm is effective in instruction-following scenarios, where the model achieves performance improvement solely through self-rewarding and iterative updates. although the self-rewarding algorithm performs well in the instruction-following tasks, it is also demonstrated that llms perform poorly on the mathematical domain data based on the existing self-rewarding algorithm. in fact, model's performance may even degrade as the number of iterations increases @xcite . we notice two main limitations in the self-rewarding framework: (a) the existing self-rewarding algorithm is not able to provide fine-grained and accurate reward signals for complex reasoning tasks involving long-thought chains; (b) for a complex mathematical solution, it's hard to design the criterion for generating specific scores. it means that assigning scores to complex long-thought multi-step reasoning for llms is more challenging than performing pairwise comparisons, with lower consistency and agreement with humans, which is proven by the results in appendix b. in this work, we propose the paradigm of process-based self-rewarding language models, where we introduce the step-wise llm-as-a-judge and step-wise preference optimization into the traditional self-rewarding framework. in a nutshell, we enable the llms to simultaneously conduct step-by-step complex reasoning and perform llm-as-a-judge for individual intermediate steps. for the limitation (a) above, to get finer-grained and more accurate rewards, process-based self-rewarding paradigm allows llms to perform stepwise llm-as-a-judge for the individual reasoning step. since producing the correct final answer does not imply that llms can generate correct intermediate reasoning steps, it is crucial to train the model to learn not only to produce the correct final answer but also to generate correct intermediate reasoning steps. by using model itself as a reward model to generate step preference pairs data, we further perform step-wise preference optimization. for the limitation (b) above, we design a llm-asa-judge prompt for step-wise pairwise comparison rather than directly assigning scores to the answer for more proper and steadier judgments based on the observations in appendix b. we conduct the experiments on models in different parameter sizes (7b and 72b) and test across a wide range of mathematical reasoning benchmarks. our results show that process-based self-rewarding can effectively enhance the mathematical reasoning capabilities of llms, which indicates that llms are able to perform effective selfrewarding at the step level. our models that iteratively trained based on the process-based self-rewarding paradigm demonstrate an increasing trend in both mathematical and llm-as-a-judge capabilities. these results suggest this framework's immense potential for achieving intelligence that may surpass human performance. ## process-based self-rewarding language models in this section, we propose our new processbased self-rewarding language models pipeline. we first review the existing self-rewarding algorithm and our motivation as a preliminary study in §3.1. then we introduce our novel paradigm for more fine-grained step-wise self-rewarding and self-evolution. the entire pipeline consists of sequential stages: model initialization ( §3.2), reasoning and preference data generation ( §3.3), and model preference optimization ( §3.4). finally, we provide a summarized overview of our algorithm ( §3.5). we illustrate the entire pipeline in figure 1 . ## experimental setup we conduct our experiments on models in different parameter sizes and several representative mathematical reasoning benchmarks. in this section, we introduce our experimental settings in detail. ## results in this section, we report our main results on different mathematical benchmarks and conduct some discussions and analyses based on the results. ## conclusion we propose a novel paradigm, process-based self-rewarding language models, that enables llms to perform step-by-step long-thought mathematical reasoning and step-wise llm-as-a-judge simultaneously. given the characteristics of complex math reasoning tasks, we introduce the step-by-step reasoning, step-wise llm-as-a-judge and step-wise preference optimization technique into the framework. our results indicate that process-based self-rewarding algorithm outperforms the original self-rewarding on a variety of complex mathematical reasoning tasks, showing potential of stronger reasoning ability better than human in the future.