Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision (2024)

Abstract: Process supervision, using a trained verifier to evaluate the intermediate steps generated by a reasoner, has demonstrated significant improvements in multi-step problem solving. In this paper, to avoid the expensive effort of human annotation on the verifier training data, we introduce Model-induced Process Supervision (MiPS), a novel method for automating data curation. MiPS annotates an intermediate step by sampling completions of this solution through the reasoning model, and obtaining an accuracy defined as the proportion of correct completions. Inaccuracies of the reasoner would cause MiPS underestimating the accuracy of intermediate steps, therefore, we suggest and empirically show that verification focusing on high predicted scores of the verifier shall be preferred over that of low predicted scores, contrary to prior observations on human curated data. Our approach significantly improves the performance of PaLM 2 on math and coding tasks (accuracy +0.67% on GSM8K, +4.16% on MATH, +0.92% on MBPP compared with an output supervision trained verifier). Additionally, our study demonstrates that the verifier exhibits strong generalization ability across different reasoning models.

## introduction multi-step problem solving (e.g., math problems and coding challenges) showcases the capabilities of machine intelligence. while researchers have shown that model-and data-upscaling still hold powerful for large language models (llms) on multi-step problem solving @xcite @xcite @xcite @xcite , even the state-of-theart llms still produce easily observable mistakes. problem: write a program for the fibonacci number. solution: def fibonacci(n):\n\tif @xmath0: def fibonacci(n): if @xmath1: return 1 elif @xmath2: return 1 return fibonacci(n -1) + fibonacci(n -2) pass completion by reasoner solution check correctness @xmath3: the model-induced process supervision (mips) data construction method we introduce in this work. by completing an intermediate solution with a reasoner several times, we can obtain the percentage value of these completions being correct. these annotations are used to train a process supervised verifier. furthermore, standard fine-tuning directly does not yield consistent and significant improvements @xcite @xcite . the reasoner-verifier paradigm (fig. 1 ) is as an inference-time technique where the goal is to pick one model-generated solution among many, since it is observed that there often are some correctly generated solutions. in particular, selfconsistency @xcite is a special case of the verifier that picks the solutions that shares the majority answer with others (e.g., math tasks where the answer is a number). llm-based verifiers are more general, as they could apply to arbitrary text solutions (e.g., code that implements a function) training a verifier in a supervised fashion has demonstrated strong performance in both coding and math language problems. @xcite showed that by simply gathering correct and incorrect solutions to train a binary classification model and using such model to pick the highest confidence solution generated by the reasoner during inference time, the accuracy can be improved significantly. more recent studies suggest that verifying on intermediate steps could offer better guidance than training solely on the whole solutions @xcite @xcite @xcite @xcite . as such, verifiers trained (and applied) on intermediate steps are called process supervised verifiers (psv), whereas those trained on whole solutions are called output supervised verifiers (osv). in prior work, process supervision data is either obtained by an ad-hoc algorithm @xcite , or through expensive human annotations @xcite , lacking an automatic and generic way of constructing of annotations of intermediate solutions. training verifier models require solution wise or step-wise labels, which is expensive to collect. there have been a series of work following an llm-as-a-verifier approach where an off-the-shelf llm is employed to judge the solutions through prompting @xcite @xcite . however, while such work may have seen improvements on language tasks, they haven't been very successful in math or coding problems @xcite . in conclusion, to achieve optimal quality, training data is needed for building a strong verifier model. on the other hand, manually collecting solution verification labels is expensive and nonscalable. in this work, we propose to use monte carlo sampling on the completions of the intermediate solutions to obtain step-wise training annotations (fig. 2 ). specifically, for each intermediate solution, we complete the solution with the reasoner several times through a sample decoding mechanism, and the percentage of the completed solutions being correct is referred to as the correctness of the solution. the correctness scores are used to train a psv. because of the nature of involving the reasoning model's completion on the intermediate solutions, we call the construction of this data model-induced process supervision (mips). while such an idea is also explored in a concurrent work @xcite , we supplement with analysis of using mips constructed data. we find that because the reasoner model, which completes the solutions, is not perfect, the noises it introduces would affect the design choices of training and using the process supervised verifier: our results reveal that (1) the verification scores from later intermediate steps are indeed useful even for a psv trained on the noisy mips data, however, the earlier step scores could hurt the verification; and (2) only using the psv predicted score of the last step, in similar fashion as osv, can sometimes be much better than osv itself, indicating process supervision data can regularize osv training. following in this paper, we will provide a more complete review of related works, a precise definition of our method, and empirical results of the method and analysis on two math problem datasets and one coding dataset. the contributions of the paper are mainly (1) we propose mips to construct process supervision data automatically for training process supervision verifiers; (2) we extend the evaluation of problem solving verifiers to coding problems; (3) we provide empirical analysis on design choices and properties of the trained verifier from mips data. ## related works the advances of problem solving of llms can be broadly characterized into two regimes, first by training a better reasoning model and the second by validating the solution from the reasoning model at inference time. pre-training/fine-tuning better reasoners. standard training recipes also transfer to training better reasoners for problem solving. during pretraining, larger model sizes and training compute yields an llm that is more competent in multiple aspects @xcite @xcite . within fine-tuning, it is also observed that transfer learning @xcite from a pile of generic math datasets, training on an augmented dataset of failure examples or diverse statements @xcite @xcite @xcite leads to improvements. despite these approaches, it is apparent that (1) the state-of-the-art llm still can fail at simple mistakes during multi-step problem solving and (2) the improvement of a simple verification method by majority voting (self-consistency @xcite ) is still significant upon fine-tuning. therefore, the exploration of verifiers to validate and pick the solutions is necessary. ## model-induced process supervision we consider the reasoner-verifier framework where we start with a fairly competent reasoner on a task, generate verifier training data on a given set of problems with the reasoner, and train a verifier on the data to validate some new generated solutions by a reasoner. we first discuss model-induced process supervision, our data curation method that automatically creates process supervision data. then, we discuss the details about the verification process. ## conclusion in this work, we introduce mips to automatically annotate intermediate solutions for multi-step problem solving. such data can be used to train a process supervised verifier that validates solutions generated by a reasoner. on two math datasets and one coding dataset, we demonstrated that mips improves the ability of picking the correct solution over an otherwise trained output supervised verifier. we conduct analysis on the aggregation function used to pick the solution and suggest that compared to verifiers trained on human-annotated process supervision, mips data trained verifiers prefer different aggregation functions. we also showed that such verifiers do not overly emphasize on the mistakes of the reasoner that produced the data, and can be transferred to different reasoners. future work could explore creating a scalable way to obtain mips data for each token in solutions to train a more competent verifier and use it to tune the reasoner via reinforcement learning.