Better Zero-Shot Reasoning with Self-Adaptive Prompting (2023)

Abstract: Modern large language models (LLMs) have demonstrated impressive capabilities at sophisticated tasks, often through step-by-step reasoning similar to humans. This is made possible by their strong few- and zero-shot abilities – they can effectively learn from a handful of handcrafted, completed responses (“in-context examples”), or are prompted to reason spontaneously through specially designed triggers. Nonetheless, some limitations have been observed. First, performance in the few-shot setting is sensitive to the choice of the examples, whose design requires significant human effort. Moreover, given the diverse downstream tasks of LLMs, it may be difficult or laborious to handcraft per-task labels. Second, while the zero-shot setting does not require handcrafting, its performance is limited due to the lack of guidance to the LLMs. To address these limitations, we propose Consistency-based Self-adaptive Prompting (COSP), a novel prompt design method for LLMs. Requiring neither handcrafted responses nor ground-truth labels, COSP selects and builds the set of examples from the LLM zero-shot outputs via carefully designed criteria combining consistency, diversity and repetition. In the zero-shot setting for three different LLMs, we show that using only LLM predictions, COSP significantly improves performance up to 15% compared to zero-shot baselines and matches or exceeds few-shot baselines at a range of reasoning tasks.

## introduction recent advances in large language models (llms) have led to state-of-the-art performance in existing natural language processing (nlp) tasks @xcite @xcite and to exciting emergent abilities @xcite . a prominent example of the latter is the strong performance on tasks that require analytical reasoning and/or methodical planning, which were previously * work done during internship at google. thought to be difficult even for massive llms @xcite . this has been made possible by scaling the model size and training corpus, the strong few-shot and zero-shot abilities of modern llms, and new techniques such as the chain-of-thought (cot) methods: few-shot cot @xcite prepends the test queries with solved input-output pairs as in-context examples to prompt the llm to generate rationales. zero-shot cot @xcite , on the other hand, appends trigger phrases after test queries to elicit reasoning. these methods constitute a competitive alternative to conventional fine-tuning which can be very costly for modern llms due to their sheer size, and is not possible when the llms are available as inference-only apis -an increasingly popular paradigm for serving models like chatgpt @xcite . despite these promising advances, numerous open challenges remain. although zero-shot cot is task-agnostic (and thus the most generally applicable in any scenario) and does not require human effort for labeling, it often underperforms its fewshot counterparts, with llms that are not shown with "template rationale" often producing spurious reasoning steps. on the other hand, for fewshot cot, the performance has been shown to be sensitive to the choice of the demonstrations @xcite , and thus improving the performance might require significant trial-and-errors and/or specific relevant expertise. given the fact that modern llms are often used for very diverse downstream tasks, selecting even a few useful per-task examples can quickly become more laborious as the number of tasks increase. this is particularly true for the cot tasks where both rationales and answers need to provided in the demonstrations. in some other cases, it could be difficult or simply impossible even to handcraft a few examples beforehand: for example, there might be tasks involving tedious and complicated reasoning or planning and/or novel test-time tasks unseen previously. in this work, our focus is to improve llm reasoning in the general zero-shot setup with access to input queries but not labels (i.e., transductive zero-shot @xcite ). we are particularly interested in this setup due to its aforementioned generality, but it is also a very challenging one given the lack of external guidance. to address this, instead of relying on simple triggers or handcrafted examples, we propose prompting the llms with the outputs generated by their own. to achieve this, we first collect a pool of rationales and answers to the test questions that are generated by the llm using zero-shot cot, and then select the most suitable questions and answers for in-context learning, similar to few-shot cot. the choice of which selfgenerated rationales to include, however, is highly non-trivial, as illustrated in fig. 1 : not only does this problem inherit all of the challenges identified above for the few-shot scenario, it also carries the additional difficulty that the self-generated demonstrations themselves are imperfect. retrieval methods assuming perfect labels @xcite are shown to perform worse than random in this setup @xcite , and previous works bypass this issue by focusing only on surface text similarity and diversity of the question embeddings @xcite , which as we show, in cases when most of the initial generated outputs are wrong, can be misleading. to address these challenges, we propose cosp (consistency-based self-adaptive prompting), an algorithm requiring only unlabelled samples (which are typically cheap to obtain, for example, via continuous, on-the-fly collections of user queries) and no labelled examples for strong performance, while also providing the capability of further improving the performance when a few labelled examples become available. the algorithm consists of two stages. in the first stage, cosp collects the pool of llm responses to test questions via zero-shot cot. we use these responses to compute the outcome entropy of each question, a metric inspired by self-consistency @xcite . this metric is then used to identify suitable question-reasoning pairs in an unsupervised way. in the second stage, we perform another pass over all test samples, but with the selected questionreasoning pairs as in-context demonstrations. we experiment with cosp with three different llms on a range of tasks. we demonstrate a 10-15% improvement in average accuracy for 6 arithhow much money does he have now? henry has 11 + 18 -10 = 27. ❌ [demo1] q: [question] a: let's think step by step. he had 11 + 18 = 29 dollars. he spent 10 dollars on a game, so he has 29 -10 = 19 dollars. ✅ [demo2] q: [question] a: let's think step by step. he had 11 dollars. he had 11 dollars. he had 11 dollars. he had 11 dollars…. ❌ [demo3] q: [question] a: let's think step by step. 11 + 18 = 29 29 -10 = 19 19 / 2 = 9 9 / 2 = 4 4 / 2 = 2 2 / 2 = 1 1 / 2 = 0 ❌ [demo3] [demo1] q: [question] a: let's think step by step. 11 + 18 = 29. 29 -10 = 19. ✅ figure 1: selecting in-context demonstrations for reasoning tasks can be a delicate art. llm output is sensitive to in-context demos and their reasoning, especially when they are generated and imperfect. example inputs & outputs shown from top to bottom (multiarith dataset & palm-62b model): (1) zero-shot cot with no demo: correct logic but wrong answer; (2) correct demo (demo1) and correct answer; (3) correct but repetitive demo (demo2) leads to repetitive outputs; (4) erroneous demo (demo3) leads to a wrong answer, but (5) combining demo3 and demo1 again leads to a correct answer. this motivates the need for a carefully-designed selection procedure for in-context demos, which is the key objective of this paper. metic and logical reasoning tasks over zero-shot cot with self-consistency baseline in palm-62b and gpt-3, @xmath0. all of these are achieved with a negligible additional computational cost. in many cases, cosp performs on par or better than the few-shot baselines with handcrafted in-context examples. chain-of-thought (cot) prompting. few-shot cot prompting @xcite prepends test questions with a series of in-context demonstrations of related solved questions as prompts. generalizing beyond the few-shot setup, kojima et al. q: a: let's think step by step. q: a: let's think step by step. q: a: let's think step by step. compute score: (eq. 8 & 9) inputs stage 1 outputs 2.0 3.0 score 2.0 3.6 3.7 2.1 3.5 1.9 q: a: since therefore the answer is 1 2 generated outputs as in-context demos llm q: a: since therefore the answer is q: a: let's think step by step. llm stage 2 outputs majority vote over outputs from both stages for nal prediction n/a j and predicted answer ŷ(i) j ) and assign a score to each. note that different rationales leading to the same final answer are marked the same color in "outputs" blocks (described in §3.2). in stage 2 (right, §3.3), we augment the current test question for which we are interested in finding in-context examples (boxes shaded in blue) with a number of in-context demonstrations. these consist of test questions and selected stage 1 outputs whose predicted answers ŷ(i) j are the majority predictions for that question that minimize the score defined in eq. ( 8 ) with the procedure described in §3.2. the augmented test question is used to query the llm a second time. finally, a majority vote over outputs from both stages forms the final prediction. self-consistency. self-consistency @xcite , which approximates the marginal distributions of the llm via majority voting of multiple decoded reasoning paths, has been shown to lead to significant performance boosts across various tasks. it is compatible with either zero-or few-shot cot by first introducing probabilistic decoding in the llm decoding using a non-zero temperature, and then sampling m reasoning paths, each consisting a rationale-answer pair (r (i) , ŷ(i) ) from the llm posterior: ## consistency-based self-adaptive prompting (cosp) as discussed in the introduction, our goal is to develop a method that improves zero-shot reasoning abilities of llms and reduces the need for human effort by automatically selecting in-context demonstrations from the llm's own generated outputs. to effectively achieve this, we propose cosp (overall procedure shown in fig. 2 and algorithm 1). in this section, we present the problem setup, followed by detailed descriptions of the two stages and the key components of the algorithm. ## related works there have been several recent works that also aim to improve llm reasoning using the models' own outputs. star @xcite . methods in the 0-shot columns use no ground-truth label guidance, whereas the 5-shot results use 5 manually labelled in-context demonstrations, randomly selected from the provided examples in @xcite . prev columns show results previously published under similar setups but often with significantly more sampling paths and/or labelled examples. "# paths" denotes the number of reasoning paths sampled per test question. cosp uses @xmath1. we also report baseline results without self-consistency (which are worse) in app. c.2. cosp and auto-cot use 5 generated in-context examples per test question. the top two results for each model are bolded and ranked by color: best and second-best. for the in-context learning setup @xcite @xcite , various methods have been proposed, typically to retrieve relevant demonstrations from a relatively large number of labelled examples @xcite . however, @xcite shows that with imperfect labels, nearest neighborstyle retrieval underperforms random selection and propose auto-cot, one of the few existing works that are fully comparable to cosp. auto-cot also uses models' zero-shot outputs as in-context demonstration by selecting the responses to the questions that are the k-means centroids in the embedding space from an auxiliary language model (a concurrent work, @xcite uses a similar clustering approach). however, by selecting based on question embedding only, auto-cot has no control over the quality of the generated rationales, and is thus prone to generating erroneous and misleading demonstrations; we empirically compare to it in §5. another concurrent work @xcite proposes self-verification, which is a zero-shot reasoning verifier to recover conditions from the predictions. however, it only works when there is a unique correct condition that results in the correct answer (such as arithmetic questions), but is not applicable when there are multiple plausible conditions. ## conclusion we propose cosp, a prompting algorithm to improve zero-shot reasoning abilities of llms. it achieves this by selecting in-context demonstrations from its own outputs using a novel scoring function that incorporates consistency, diversity and repetitiveness. across multiple llms and tasks requiring complex reassigning, we demonstrate large improvements over the state of the art. we believe there are multiple promising directions for future work: first, the present work focuses on improving zero-shot reasoning, but we argue that the key principle used in this paper may also be extended to more general nlp task types. second, beyond solving reasoning problems per se, llms' reasoning capabilities are increasingly used in other scenarios for planning & interactions with external environments @xcite -extending cosp to these setups could also be promising. we defer thorough investigations to a future work.