Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios (2024)

Abstract: Although chain-of-thought (CoT) prompting combined with language models has achieved encouraging results on complex reasoning tasks, the naive greedy decoding used in CoT prompting usually causes the repetitiveness and local optimality. To address this shortcoming, ensemble-optimization tries to obtain multiple reasoning paths to get the final answer assembly. However, current ensemble-optimization methods either simply employ rule-based post-processing such as self-consistency, or train an additional model based on several task-related human annotations to select the best one among multiple reasoning paths, yet fail to generalize to realistic settings where the type of input questions is unknown or the answer format of reasoning paths is unknown. To avoid their limitations, we propose Self-Agreement, a generalizable ensemble-optimization method applying in almost all scenarios where the type of input questions and the answer format of reasoning paths may be known or unknown. Self-agreement firstly samples from language model’s decoder to generate a diverse set of reasoning paths, and subsequently prompts the language model one more time to determine the optimal answer by selecting the most agreed answer among the sampled reasoning paths. Self-agreement simultaneously achieves remarkable performance on six public reasoning benchmarks and superior generalization capabilities.

## introduction although large language models (llms) have revolutionized the natural language processing (nlp) landscape, their ability to solve challenging tasks (e.g., arithmetic, commonsense and symbolic reasoning) is often seen as a limitation, which is difficult to be overcome solely by scaling up the size of llms @xcite . to address this issue, @xcite have proposed chain-of-thought (cot) prompting, which this work studies ensemble-optimization @xcite in multi-step reasoning situations. ensemble-optimization tries to obtain multiple reasoning paths to get the final answer assembly. it avoids the repetitiveness and local optimality that plague greedy decoding, while mitigating the stochasticity caused by a single sampled generation @xcite . current ensemble-optimization methods predominantly fall into two categories, i.e., verifier or re-ranker based methods and post-processing based methods. verifier or re-ranker based methods either train an additional verifier @xcite or train a re-ranker based on task-related human annotations @xcite to select the best generation among multiple generations. post-processing based methods take a majority vote among all generated reasoning paths @xcite @xcite or top k complex reasoning paths @xcite , and then choose the optimal answer that receives the most votes. while the success of ensemble-optimization work, we show that there are two major limitations. figure 1 illustrates the main limitations of existing ensemble-optimization methods. on one hand, although verifier or re-ranker based methods have favorable generalization ability to some extent as they impose no restrictions on answer formats of reasoning paths, but they can only be applied in scenario where the type of questions is already known, yet fail to generalize to scenarios where the ## related work in-context learning language models have revolutionized a wide range of nlp tasks, where scaling up the model size is one of the key ingredients @xcite @xcite @xcite @xcite . the success of llms is often attributed to emergent abilities when the model reaches a sufficient scale @xcite . that is, the model can follow the format of given prompts (typically a few task-specific examples) thus solving the corresponding tasks (also referred as in-context learning). the method of conditioning llms is called "prompting" @xcite , which can be categorized into two main directions: few-shot prompting (a few examples as the prompt) 2 after completion of this work, we find that @xcite share almost the same idea with us, which is archived after our work. we provide discussion with it in appendix d. and zero-shot prompting (instructions describing the task as the prompt). prompting allows a single model to carry out various tasks universally. due to its superior benefits, there are studies @xcite @xcite further investigate how to improve the performance of in-context learning. specifically, different wording or order of given examples may lead to performance fluctuations @xcite . this work takes an important step forward in multi-step reasoning by showing the critical role of language model itself in the process of selecting the most agreed answer based on multiple reasoning paths. reasoning with language models reasoning, the process of making inference based on existing information or knowledge, is the core of human intelligence and essential for solving complex questions @xcite . in contrast to the excellent performance of llms in simple and single-step tasks, language models (even 100b or more parameters) are demonstrated to struggle at solving challenging tasks required multi-step reasoning @xcite . to address this issue, @xcite have proposed to decompose multi-step reasoning problems into intermediate steps before obtaining final answers. furthermore, @xcite have proposed chainof-thought prompting, which elicits this reasoning process from language models. since then, designing prompts manually @xcite @xcite or automatically @xcite @xcite , example selection in cot prompting @xcite @xcite , compositional reasoning @xcite and data augmentation @xcite @xcite have become a hot topic in nlp. our work sits in the context of zero-shot cot and few-shot cot, and proposes a language model (1) ask k times q: james runs 12 miles a day for 5 days a week. if he runs 10 miles an hour how many hours does he run a week? gsm8k q: what island country is ferret popular? answer choices: (a) own home (b) north carolina (c) great britain (d) hutch (e) outdoors csqa or ...... 1. first, we can calculate how many miles james runs in a week: \n\n12 miles/day x 5, {...}, @xmath0. step 1: calculate the total distance james runs in a week. \n12 miles per day * 5 days per @xmath1, {...}, james runs for 6 hours a week. first, we can calculate how many miles james runs in a week by multiplying the number of miles, {...}, james runs for a total of 6 hours. sample diverse reasoning paths when making a majority vote on multiple corresponding solutions, {...}, follow these detailed steps: 1.understand the question: {...}; 2.extract the final answer for each solution: {...}; 3.count the votes: {...}; 4.select the final answer: {...}. language model (2) ask one more time ### question: james runs 12 miles a day, {...}? solution 1: 1. first, we can calculate how, {...}. solution 2: step 1: calculate the total, {...}. solution 3: first, we can calculate how many, {...}. ### ...... the majority voted answer is: james runs for 6 hours a week. figure 3 : the self-agreement method contains two stages: (1) ask k times: sample from the language model itself k times to generate k diverse reasoning paths using few-shot cot @xcite or zero-shot cot @xcite ; and (2) ask one more time: select the most agreed answer based on k reasoning paths by language model itself. yellow denotes the carefully designed prompt proposed in this work (see ## methodology existing ensemble-optimization studies @xcite commonly assume that both the type of questions fed to the model and answer format of reasoning paths outputted by the model are already known, and conduct evaluations on the questions from the same dataset. however, a more realistic setting is that the type of input questions or answer format of reasoning paths is unknown and they come in an arbitrary manner. to address such scenarios, one natural idea is that can we allow the language model itself to select the best generation among multiple generations, since it has favorable generalization ability, i.e., no restrictions on the types of questions and answer formats of reasoning paths, and avoids any additional training, auxiliary models or fine-tuning. beforehand, we need to figure out what language models are good at for? previous work has demonstrated that language models are good answer extractors given specific prompts @xcite @xcite , and also suitable for comparing the consistency of final answers extracted from multiple reasoning paths given specific prompts @xcite , as the extracted final answers in reasoning tasks tend to be short and semantically straightforward sentences or phrases. for instance, as shown in figure 3 , for language models, james runs for 6 hours a week and james runs for a total of 6 hours extracted from solution 2, 3 respectively represent the same final answer. such behaviors are often attributed to emergent abilities @xcite . that is, the model can understand and follow the format of given prompts thus solving the corresponding tasks. motivated by the above findings and the majority vote design used in @xcite , we propose the following self-agreement method that allows language model itself to extract final answers of multiple reasoning paths, count the votes of different answers, and then select the majority voted answer. specifically, firstly, we prompt the language model with few-shot cot @xcite (w/ answer trigger) or zero-shot cot @xcite (wo/ answer trigger). then, we sample from language model's decoder to generate a diverse set of reasoning paths. we refer to this stage as ask ## experiments we carry out a series of experiments to confirm the efficacy of our method on three scenarios. our findings indicate that across a wide range of tasks, scenarios, models, and prompting methods, selfagreement generally enhances the reasoning performance of language models. we introduce experimental setup in §4.1, main results in §4.2, and analysis in §4.3. see appendix a and b for more experimental details and additional experiments. ## conclusion in this work, we explore a more realistic setting with significant application values in ensembleoptimization studies, as shown in figure 2 . to this end, we propose self-agreement, a simple yet generalizable ensemble-optimization method applying in almost all scenarios. self-agreement simultaneously achieves remarkable performances on six public reasoning benchmarks and superior generalization capabilities. our findings encourage the research community to focus on a deeper understanding of the role of language model itself in ensemble-optimization studies, which has favorable generalization abilities and superior performance. we hope this work will open new research possibilities in prompting, language models, ensemble-optimization and multi-step reasoning.