Math-Shepherd: Verify and Reinforce LLM s Step-by-step without Human Annotations (2024)

Abstract: In this paper, we present an innovative process-oriented math process reward model called Math-shepherd, which assigns a reward score to each step of math problem solutions. The training of Math-shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-shepherd in two scenarios: 1) \textit{Verification} : Math-shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) \textit{Reinforcement Learning (RL)} : Math-shepherd is employed to reinforce LLMs.With Math-shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, process RL with Math-shepherd significantly enhances Mistral-7B (77.9% \to 84.1% on GSM8K and 28.6% \to 33.0% on MATH).The accuracy can be further improved to 89.1% and 43.5% on two benchmarks with verification of Math-shepherd.We believe that automatic process supervision holds significant potential for the future evolution of LLMs.

## introduction large language models (llms) have demonstrated remarkable capabilities across various tasks @xcite @xcite @xcite @xcite , however, even the most advanced llms face challenges in complex multi-step mathematical reasoning problems @xcite . to address this issue, prior research has explored different methodologies, such as pre-training @xcite , fine-tuning @xcite @xcite , prompting @xcite , and verification @xcite @xcite . among these tech-niques, verification has recently emerged as a favored method. the motivation behind verification is that relying solely on the top-1 result may not always produce reliable outcomes. a verification model can rerank candidate responses, ensuring higher accuracy and consistency in the outputs of llms. in addition, a good verification model can also offer invaluable feedback for further improvement of llms @xcite @xcite . the verification models generally fall into the outcome reward model (orm) @xcite and process reward model (prm) @xcite @xcite @xcite . the orm assigns a confidence score based on the entire generation sequence, whereas the prm evaluates the reasoning path step-by-step. prm is advantageous due to several compelling reasons. one major benefit is its ability to offer precise feedback by identifying the specific location of any errors that may arise, which is a valuable signal in reinforcement learning and automatic correction. besides, the prm exhibits similarities to human behavior when assessing a reasoning problem. if any steps contain an error, the final result is more likely to be incorrect, mirroring the way human judgment works. however, gathering data to train a prm can be an arduous process. @xcite and @xcite utilize human annotators to provide process supervision annotations, enhancing the performance of prm. nevertheless, annotation by humans, particularly for intricate multi-step reasoning tasks that require advanced annotator skills, can be quite costly, which hinders the advancement and practical application of prm. to tackle the problem, in this paper, we propose an automatic process annotation framework. inspired by monte carlo tree search @xcite @xcite , we define the quality of an intermediate step as its potential to deduce the correct final answer. by leveraging the correctness of the answer, we can automatically gather step-wise supervision. specifically, given a math problem with a golden answer and a stepby-step solution, to achieve the label of a specific step, we utilize a fine-tuned llm to decode multiple subsequent reasoning paths from this step. we further validate whether the decoded final answer matches with the golden answer. if a reasoning step can deduce more correct answers than another, it would be assigned a higher correctness score. we use this automatic way to construct the training data for math-shepherd, and verify our ideas on two widely used mathematical benchmarks, gsm8k @xcite and math @xcite . we explore the effectiveness of math-shepherd in two scenarios: 1) verification: math-shepherd is utilized for reranking multiple outputs generated by llms; 2) reinforcement learning: math-shepherd is employed to reinforce llms with step-by-step ppo. with the verification of math-shepherd, a series of open-source llms from 7b to 70b demonstrates exceptional performance. for instance, the step-bystep ppo with math-shepherd significantly improves the accuracy of mistral-7b (77.9%→84.1% on gsm8k and 28.6%→33.0% on math). the accuracy can be further enhanced to 89.1% and 43.5% on gsm8k and math with verification. deepseek 67b @xcite achieves accuracy rates of 93.3% on the gsm8k dataset and 48.1% on the math dataset with verification of math-shepherd. to the best of our knowledge, these results are unprecedented for open-source models that do not rely on additional tools. our main contributions are as follows: 1) we propose a framework to automatically construct process supervision datasets without human annotations for math reasoning tasks; 2) we evaluate our method on both step-by-step verification and reinforcement learning scenarios. extensive experiments on two widely used mathematical benchmarks -gsm8k and math, in addition to a series of llms ranging from 7b to 70b, demonstrate the effectiveness of our method; 3) we empirically analyze the key factors for training high-performing process reward models, shedding light on future directions toward improving reasoning capability with automatic step-by-step verification and supervision. ## methodology in this section, we first present our task formulation to evaluate the performance of reward models ( §3.1). subsequently, we outline two typical categories of reward models, orm and prm( §3.2). then, we introduce our methodology to automatically build the training dataset for prm( §3.3), breaking the bottleneck of heavy reliance on manual annotation in existing work @xcite . ## experiments datasets we conduct our experiments using two widely used math reasoning datasets, gsm8k @xcite and math @xcite . for the gsm8k dataset, we leverage the whole test set in both verification and reinforcement learning scenarios. for the math dataset, in the verification scenario, due to the computation cost, we employ a subset math500 that is identical to the test set of @xcite . the subset consists of 500 representative problems, and we find that the subset evaluation produces similar results to the full-set evaluation. to assess different verification methods, we generate 256 candidate solutions for each test problem. we report the mean accuracy of 3 groups of sampling results. in the reinforcement learning scenario, we use the whole test set to evaluate the model performance. we train llms with metamath @xcite . ## conclusion in this paper, we introduce a process-oriented math verifier called math-shepherd, which assigns a reward score to each step of the llm's outputs on math problems. the training of math-shepherd is achieved using automatically constructed process-wise supervision data, thereby eradicating the necessity for labor-intensive human annotation. remarkably, this automatic methodology correlates strongly with human annotations. extensive experiments in both verification and reinforcement learning scenarios demonstrate the effectiveness of our method.