Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data (2023)

Abstract: Chain-of-thought (CoT) advances the reasoning abilities of large language models (LLMs) and achieves superior performance in complex reasoning tasks. However, most CoT studies rely on carefully designed human-annotated rational chains to prompt LLMs, posing challenges for real-world applications where labeled data is available without rational chains. This paper proposes a new strategy, AutomateCoT (Automatic Prompt Augmentation and Selection with Chain-of-Thought), that can bypass human engineering of CoT by automatically augmenting rational chains from a small labeled dataset, and then pruning low-quality chains to construct a candidate pool of machinegenerated rationale chains based on the labels. Finally, it selects the optimal combination of several rationale chains from the pool for CoT prompting by employing a variance-reduced policy gradient strategy to estimate the significance of each example. Automate-CoT enables a quick adaptation of the CoT technique to different tasks. Experimental results demonstrate the effectiveness of our method, where competitive results are achieved on arithmetic reasoning (+2.7%), commonsense reasoning (+3.4%), symbolic reasoning (+3.2%), and non-reasoning tasks (+2.5%).

## introduction the recent success in large language models (llms) has shown that properly prompted llms demonstrate emergent capabilities on complex understanding and question-answering tasks @xcite . especially, with the recently proposed chain-of-thought (cot) prompting @xcite , llms are capable of solving reasoning tasks including arithmetic reasoning, commonsense reasoning, and symbolic reasoning. the basic idea of cot prompting is adding a few rationale chains to the answer as exemplars to illustrate the intermediate reasoning steps. following cot, several recent studies improve it by leveraging self-consistency @xcite , explanation learning @xcite , complexity-based prompting @xcite , self-training @xcite , voting verifier @xcite , and bootstrapping @xcite . however, most of them are constrained to a few fixed human-written exemplars, which require significant human efforts to create and adapt to new datasets. the annotation process is nontrivial because humans need to not only select the questions but also carefully design the reasoning steps for each question. in the process of searching for the perfect exemplars, we identify four critical factors that affect the performance of chain-of-thought prompting and require large human effort to deal with: (1) order sensitivity @xcite : the order combination of the exemplars; (2) complexity @xcite @xcite : the number of reasoning steps of the rationale chains; (3) diversity: the combination of different complex-level exemplars; (4) style sensitivity @xcite : the writing/linguistic style of the rationale chains. detailed analysis of the four factors is covered in section 2. all of these sensitivities make human-based prompt engineering costly and motivate us to find an automatic and task-agnostic way to adapt chain-ofthought exemplars to any downstream tasks. in this paper, we solve these problems by a cot augmentation and selection process to find suitable exemplars automatically. this can be divided into three steps: (1) augment: the language model generates multiple pseudo-chains for query questions automatically. (2) prune: based on an assumption: generating correct reasoning is a necessary condition for generating correct answers. this assumption is natural because the answer is generated after several reasoning steps. when a correct answer is generated, the rationale chain of these steps is most likely correct, contributing to the final correctness. as a result, we prune the pseudo-chains according to the consistency between generated and ground-truth answers to reduce the noise. (3) select: given that all the data have been annotated with rationale paths, we propose to apply a variance-reduced policy gradient strategy @xcite @xcite to estimate the gradients and optimize the selection process to find the most helpful chain-of-thought for each task. compared to prior manually written cot, automate-cot could find the optimal and diverse cot automatically, adaptable to any task without human effort. compared with auto-cot @xcite , which samples diverse questions by clustering and generates rationale chains, automate-cot considers and mitigates the aforementioned sensitivity issues, while achieving a greater performance boost for each task. automate-cot is a fully automatic pipeline for finding better chainof-thought prompts, mitigating the sensitivity issues of manually written exemplars, and further improving the performance by a large margin. experimental results demonstrate the effectiveness of automate-cot on arithmetic reasoning (+2.7%), commonsense reasoning (+3.4%), symbolic reasoning (+3.2%), and non-reasoning tasks (+2.5%). ## motivation recent studies observed sensitivity issues of gpt-3's few-shot learning caused by different selections of in-context examples such as order instability @xcite @xcite . based on their findings, we first investigate whether these sensitivities still exist in chain-of-thought methods. then we further explore other factors that would not only affect the performance but require human efforts to deal with. we conclude with the following four factors: ## approach our approach receives a training dataset d containing n questions @xmath0, q 2 , ..., q n }, and n answers @xmath1, a 2 , ..., a n }. the overall architecture of our approach is shown in figure 2 . in this section, we start with a detailed description of augment and prune operation and end with an illustration of select operation. ## experimental settings in this section, we first introduce the setting of eleven datasets and their corresponding evaluation metrics ( § 4.1). then the baseline models ( § 4.2) and implementation details ( § 4.3) are presented in the following two subsections, respectively. full details about the experimental setting are illustrated in appendix b. ## experimental results the experimental results are shown in cot also shows similar trends to improve the sc baseline, demonstrating the synergistic effects of our proposed method and self-consistency method. commonsense and symbolic reasoning similarly, on commonsense and symbolic reasoning tasks, automate-cot demonstrates significant improvement over manual-cot, sc, and auto-cot. it achieves an average of 2.5% and 3.4% improvement on text-davinci-002 and code-davinci-002 respectively, demonstrating that our method is effective on different task types. more surprisingly, the improvement in the letter (4) is significant, demonstrating our method's robustness to deal with out-of-distribution data. non-reasoning tasks automate-cot has also reached great success on question answering (openbookqa), natural language inference (e-snli), and sentiment analysis (sst-2) tasks by an improvement of 2.8%, 3.4% and 1.3%, respectively. the results show that our method can be generalized to various task types and is not limited to reasoning tasks. ## additional experiments and analysis we further conduct several experiments to evaluate the effectiveness of automate-cot and analyze the contributions of each module. since queries to text-davinci-002 are limited and expensive, most additional experiments are conducted with code-davinci-002. ## ablation study in this section, we further conduct ablation experiments to verify the advantage of the generated prompts on four factors, respectively. ## conclusion in this paper, we proposed a chain-of-thought optimization method consisting of three steps: augment, prune, and select. automate-cot first generates rationale chains according to the standard cot process with several exemplars, and then prunes those incorrect ones according to the consistency of the predicted answer and ground-truth answer. finally, we apply a variance-reduced policy gradient strategy to estimate the gradients and optimize the latent variables to select better cots. experimental results demonstrate the effectiveness of our method on arithmetic reasoning, commonsense reasoning, symbolic reasoning tasks, and non-reasoning tasks.