Making Language Models Better Reasoners with Step-Aware Verifier (2023)

Abstract: Few-shot learning is a challenging task that requires language models to generalize from limited examples. Large language models like GPT-3 and PaLM have made impressive progress in this area, but they still face difficulties in reasoning tasks such as GSM8K, a benchmark for arithmetic problems. To improve their reasoning skills, previous work has proposed to guide the language model with prompts that elicit a series of reasoning steps before giving the final answer, achieving a significant improvement on GSM8K from 17.9% to 58.1% in problem-solving rate. In this paper, we present DiVeRSe (Diverse Verifier on Reasoning Step), a novel approach that further enhances the reasoning capability of language models. DiVeRSe has three main components: first, it generates diverse prompts to explore different reasoning paths for the same question; second, it uses a verifier to filter out incorrect answers based on a weighted voting scheme; and third, it verifies each reasoning step individually instead of the whole chain. We evaluate DiVeRSe on the latest language model code-davinci-002 and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks (e.g., GSM8K 74.4% to 83.2%).

## introduction large pretrained language models (plms) have shown remarkable performance on various natural language processing tasks, either by few-shot learning with prompts @xcite @xcite or by fine-tuning @xcite @xcite . however, despite the increasing size and capacity of plms such as gpt-3 with 175b parameters @xcite and palm with 540b parameters @xcite , their reasoning abilities are still limited and often require mul- * work was done during an internship at microsoft research asia. tiple steps to produce correct answers, especially for tasks involving arithmetic, commonsense, or inductive reasoning @xcite . recent works @xcite @xcite have demonstrated that plms possess some latent reasoning capabilities, but they need carefully designed prompts to activate them. for instance, @xcite proposed chain-of-thought reasoning, which inserts multi-step reasoning paths before generating the final answers, and achieved significant improvement on the gsm8k arithmetic benchmark @xcite . @xcite further introduced a voting mechanism to select the most consistent answer among different reasoning paths, and achieved state-of-the-art results on several reasoning benchmarks using the palm model @xcite . building on these successes, this paper continues this line of research and advances the reasoning capabilities of plms in three aspects, as illustrated in figure 1 . first, we propose to increase the diversity of reasoning paths by not only sampling from a single prompt, but also varying the prompt itself. we hypothesize that different prompts can elicit different ways of thinking, while the correct answer should be robust to these variations. second, we propose to use a verifier to score the quality of each reasoning path and guide the voting mechanism. we argue that not all reasoning paths are equally good chain-of-thought reasoning for gsm8k math word problem q: if there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot? a: there are 3 cars in the parking lot already. 2 more arrive. now there are 3 + 2 = 5 cars. the answer is 5. ... q: janet's ducks lay 16 eggs per day. she eats three for breakfast every morning and bakes muffins for her friends every day with four. she sells the remainder for $2 per egg. how much does she make every day? a: she has 16 -3 -4 = 9 eggs left. so she makes 2 * 9 =18 per day. the answer is 18. figure 2 : chain-of-thought reasoning for gsm8k math word problem. the prompt is colored black and the reasoning path produced by the language model is colored teal. this reasoning path contains two reasoning steps. or reliable, and some may contain errors or inconsistencies that can be detected by the verifier. third, we propose to assign a fine-grained label to each step of the reasoning path and use a step-aware verifier to attribute the correctness or wrongness of the final answer to each step. we conjecture that some steps may be correct but followed by wrong steps or vice versa, and identifying these cases can help diagnose and improve the reasoning process. we name our method as diverse (diverse verifier on reasoning step) and evaluate it on eight reasoning benchmarks that require different types of reasoning skills. we use three openai plms (davinci, text-davinci-002, and code-davinci-002) and compare our results with recent state-of-the-art methods. we find that diverse can consistently and significantly improve the performance of plms on these tasks, and achieve new state-of-the-art results on six of them 1 : gsm8k (74.4% → 83.2%), asdiv (81.9% → 88.7%), multiarith (99.3% → 99.8%), svamp(86.6% → 87.0%), singleeq (79.5% → 94.9%), and clutrr (67.0% → 95.9%). our data is publicly available at https://github. com/microsoft/diverse . ## diverse verifier on reasoning step figure 1 shows the overview of diverse. the key insights are three-fold: (1) leveraging diverse prompts to induce more diverse reasoning paths from the language models (section 2.1); (2) train-1 most of the previous sota results were achieved by selfconsistency on palm-540b @xcite . ## main results ## analysis we also conduct ablation experiments and analysis to investigate the keys to the success of diverse. ## related work reasoning skills. researchers in the literature have proposed many benchmarks requiring various reasoning skills, including commonsense reasoning @xcite @xcite numerical reasoning @xcite , multi-hop reasoning @xcite , arithmetic reasoning @xcite @xcite @xcite , logical reasoning @xcite , inductive reasoning @xcite and tabular reasoning @xcite . reasoning with symbolic systems. much research in the literature enhances the reasoning capabilities of machine learning systems by exploiting symbolic systems, including knowledge graphs @xcite @xcite @xcite @xcite , or question taxonomies @xcite @xcite . although these methods work well on specific benchmarks, they usually require domain-specific designs and human efforts, thus limiting the generalizability. reasoning via language models. this line of work aims to address reasoning tasks in a general sequence-to-sequence manner, empowered by reasoning-aware pre-training or fine-tuning of language models. for example, deng et al. (2021) 0 250 500 1000 sampled (or generated) training data size 50 55 60 65 70 75 80 solve rate (%) gsm8k voting verifier voting verifier figure 8: diverse performance (code-davinci-002) on gsm8k with different sizes of the training dataset (without labeled reasoning paths). proposed to train the language model with crawled data from the internet; asai and hajishirzi (2020) proposed a logic-guided data augmentation method to pre-train the language model; shen et al. (2021); cobbe et al. (2021) proposed to train a verifier to rank solutions sampled from fine-tuned language models; geva et al. (2020); yoran et al. (2022); campagna et al. (2020); wang et al. (2022a) proposed to equip language models with reasoning abilities by generating training examples with human-designed templates; pi et al. (2022) proposed to inject reasoning capabilities into language models by continual pre-training on program execution data. reasoning via prompting gigantic language models. gigantic language models like gpt-3 @xcite have demonstrated impressive few-shot learning capabilities in many tasks and have attracted many research interests on making gigantic language models better few-shot learners @xcite @xcite @xcite @xcite . however, these methods struggle to address tasks requiring reasoning skills. to mitigate this, recently there is a line of research that focuses on unleashing the reasoning capabilities of gigantic language models via better prompting strategies. wei et al. (2022) proposed chainof-thought reasoning, of which the key insight is the insertion of multi-step reasoning paths before generating the final answers; wang et al. (2022c) proposed to improve chain-of-thought reasoning via self-consistency, of which the key insight is to conclude the most consistent answer from different reasoning paths sampled from the language model; zhou et al. (2022); creswell et al. (2022) 3 5 8 number of exemplars in one prompt 50 55 60 65 70 75 80 85 solve rate (%) 51.8 55 56 79.5 80 79.6 82 82.3 83.2 gsm8k greedy decode divprompts+voting diverse proposed to leverage gigantic language models to decompose questions into sub-questions, thereby addressing them in an iterative manner; @xcite proposed that gigantic language models can even be good zero-shot reasoners, by designing prompts that can induce language models to do reasoning step-by-step; lampinen et al. (2022) proposed building a prompt by selecting examples and explanations together, thus substantially improving performance over selecting examples alone. despite their great successes, these works come with their limitations. this paper is a continuation of this line of research, focusing on diverse verifier on reasoning steps. ## conclusion and future work in this paper, we present diverse, a novel and general method to enhance the reasoning abilities of large language models. our method builds on the idea of prompting language models with multistep reasoning paths, but introduces three key innovations: diverse prompts, voting verifier, and stepwise verifier. the latter is especially novel and effective, as it verifies each reasoning step separately and we provides a detailed analysis of the model's behavior in each step. we demonstrate the superiority of diverse through extensive experiments. for instance, using code-davinci-002, our method achieves state-of-the-art performance on most reasoning tasks, surpassing the 540b palm model with previous prompting techniques. there are many directions for our future work. (1) as discussed in appendix b.2, we will continue to investigate how to reduce or recognize false positive pseudo exemplars. (2) we plan to investigate mechanisms to produce better diverse prompts than simple sampling. (3) we will extend diverse to other tasks and continue to design better prompting techniques to elicit the power of gigantic language models.