Step-level Value Preference Optimization for Mathematical Reasoning (2024)

Abstract: Direct Preference Optimization (DPO) using an implicit reward model has proven to be an effective alternative to reinforcement learning from human feedback (RLHF) for fine-tuning preference aligned large language models (LLMs). However, the overall preference annotations of responses do not fully capture the fine-grained quality of model outputs in complex multi-step reasoning tasks, such as mathematical reasoning. To address this limitation, we introduce a novel algorithm called Step-level Value Preference Optimization (SVPO). Our approach employs Monte Carlo Tree Search (MCTS) to automatically annotate step-level preferences for multi-step reasoning. Furthermore, from the perspective of learning-to-rank, we train an explicit value model to replicate the behavior of the implicit reward model, complementing standard preference optimization. This value model enables the LLM to generate higher reward responses with minimal cost during inference. Experimental results demonstrate that our method achieves state-of-the-art performance on both in-domain and out-of-domain mathematical reasoning benchmarks.

## introduction recently, large language models (llms) have demonstrated remarkable capability across a wide range of natural language processing (nlp) tasks @xcite @xcite @xcite @xcite . however, they continue to encounter significant challenges when engaging in complex and symbolic multi-step reasoning, particularly in mathematical reasoning @xcite @xcite @xcite @xcite . most existing studies @xcite @xcite @xcite have significantly improved the mathematical reasoning capabilities through fine-tuning on high-quality positive supervision data (i.e, correct solutions) annotated by gpt-4. in this process, a large number of negative examples generated by gpt-4 are wasted, and the model blindly imitates successful cases without understanding what the wrong solutions are. therefore, preference learning, such as direct preference optimization (dpo) @xcite , has been proposed to align with human preferences and enable the model to distinguish between positive and negative examples. however, most current efforts @xcite @xcite focus on solution-level preferences, relying on humans or gpt-4 to generate and score complete solutions for training. this approach is expensive and often provides only a coarse preference relationship, which does not reflect the natural process by which humans learn to solve mathematical problems. this discrepancy arises because solution-level preferences pursue a solution to its final answer, without informing which step in the negative solution (i.e., y l ) led to the mistake. unlike these approaches, humans tend to identify and analyze their mistakes step by step when learning to solve mathematical problems, thereby preventing repeated errors. in this manner, humans progressively learn to make informed decisions in similar states. furthermore, while dpo reparameterizes the reward function in reinforcement learning from hu-man feedback (rlhf) @xcite to improve simplicity and training stability, it also discards the state-value function v (s), which is used to evaluate the expected return from the current state. recent work @xcite has demonstrated the effectiveness of the value model in improving the reasoning capabilities of policy models, but it is limited by the need for additional annotated data or the complexity of the reinforcement learning process. to address the above issues, we propose steplevel value preference optimization (svpo), a novel preference learning framework that focuses on more fine-grained step-level preferences via monte carlo tree search (mcts) to significantly enhance mathematical reasoning capabilities. specifically, as illustrated in figure 1 , step-level preferences are autonomously generated through the mcts framework @xcite . this approach not only avoids labor-intensive annotation but also provides detailed insights into which steps may lead to mistakes in y l , as indicated by the q-value at each node. compared to the forced knowledge infusion through gpt-4 annotated data, the preferences obtained through self-exploration are better aligned with the capabilities of the current llm, highlighting the reasoning errors that the model is more prone to making. furthermore, we integrate an explicit value model with dpo, where the value model is designed not only to assist the policy model (i.e., llm) in navigating more effective reasoning paths but also to steer preference learning. in our work, the value model is trained based on both q-values and step-level preference relationships derived from mcts, thereby bypassing the need for additional annotations and simplifying the training process. we conduct extensive experiments on both indomain and out-of-domain mathematical reasoning datasets. our svpo significantly outperforms stateof-the-art methods, achieving comparable or even superior results to gpt-4 on 7b llms. the experiments demonstrate three key points: (1), the self-exploration process via mcts naturally provides step-level preference relationships and highlights potential reasoning errors by q-values; (2), compared to solution-level preferences, step-level preferences can significantly enhance the reasoning capabilities of the policy model; (3), the value model effectively guides the policy model's preference learning and reasoning. ## background in standard rlhf framework, it first learns a reward model r(x, y) with bradley-terry (bradley and terry, 1952) preference optimization. ## method in this section, we present our svpo in detail to further explore the potential of preference learning in multi-step reasoning tasks, particularly in mathematical reasoning.