Active Prompting with Chain-of-Thought for Large Language Models (2024)

Abstract: The increasing scale of large language models (LLMs) brings emergent abilities to various complex tasks requiring reasoning, such as arithmetic and commonsense reasoning. It is known that the effective design of task-specific prompts is critical for LLMs’ ability to produce high-quality answers. In particular, an effective approach for complex question-and-answering tasks is example-based prompting with chain-of-thought (CoT) reasoning, which significantly improves the performance of LLMs. However, current CoT methods rely on a fixed set of human-annotated exemplars, which are not necessarily the most effective examples for different tasks. This paper proposes a new method, Active-Prompt, to adapt LLMs to different tasks with task-specific example prompts (annotated with human-designed CoT reasoning). For this purpose, we propose a solution to the key problem of determining which questions are the most important and helpful to annotate from a pool of task-specific queries. By borrowing ideas from the related problem of uncertainty-based active learning, we introduce several metrics to characterize the uncertainty so as to select the most uncertain questions for annotation. Experimental results demonstrate the superiority of our proposed method, achieving superior performance on eight complex reasoning tasks. Further analyses of different uncertainty metrics, pool sizes, zero-shot learning, and accuracy-uncertainty relationships demonstrate the effectiveness of our method.

## introduction large language models (llms) @xcite @xcite @xcite @xcite have achieved great success in recent years. a typical way of applying llms is in-context learning (brown et al., 1 our code is available at https://github.com/ shizhediao/active-prompt . 2020) by providing a number of instructions and exemplars, which performs well on conventional language understanding and generation tasks but performs poorly on complex reasoning tasks @xcite @xcite . recent prompting studies @xcite @xcite found that elaborating the reasoning steps in the exemplars endows llms with good reasoning abilities, namely chainof-thought (cot) prompting. however, chain-ofthought prompting depends on human engineering: it requires humans to select a few informative questions and then annotate them with cot and answers. the human-annotated exemplars (questions with annotated cot and answers) are not necessarily the most effective for different tasks. for example, the original chain-of-thought prompting @xcite crafted exemplars for eight questions, which are either randomly selected from the training set or manually composed by humans. due to there being a significant variance in the nature of reasoning tasks in terms of difficulty, scope, domain, and so on, we do not know what kind of question is the most worthy of annotating. it is also not clear whether a particular set of exemplars is the best to elicit the desired information. however, the good news is that annotating eight exemplars for different tasks is trivial. it costs little money and human effort. in light of this, we identify the key problem as how to determine which questions are the most important and helpful for annotation. we propose a solution to this problem by leveraging uncertainty and introducing a bit of human effort to annotate a small set of questions. the annotation budget is reasonable. by borrowing ideas from the related problem of uncertainty-based active learning @xcite , we introduce several metrics to characterize the uncertainty among the model's predictions on each question. therefore, we propose a new uncertainty-based annotation strategy that chooses (1) uncertainty estimation: with or without a few human-written chain-of-thoughts, we query the large language model k (k " 5 in this illustration) times to generate possible answers with intermediate steps for a set of training questions. then, we calculate the uncertainty u based on k answers via an uncertainty metric (we use disagreement in this illustration). (2) selection: according to the uncertainty, we select the most uncertain questions for annotation. (3) annotation: we involve humans to annotate the selected questions. (4) inference: infer each question with the new annotated exemplars. a number of questions from the downstream dataset and involves humans annotating the rational chains, significantly improving the performance. specifically, given a dataset d, we first ask the model to answer it k times. then, we calculate the uncertainty u of this model based on k answers to each question. with u, we select the most uncertain n questions with the largest u and annotate these questions by the oracle to craft new exemplars e. finally, we pre-pend e to each test question following the standard recipe of chain-of-thought prompting @xcite . the schematics of our proposed approach are illustrated in figure 1 . there are several different ways for uncertainty estimation in the literature @xcite . in our main experiments, we characterize the uncertainty u by the disagreement and entropy of all predicted answers. in addition, we investigate other different uncertainty metrics, like variance and self-confidence. for selfconfidence, we re-organize the generated answer with the question using a new template and then ask the model's confidence for such generation. in this scenario, u is defined as a categorical variable from {very confident, confident, not confident, wrong answer}. it is observed that the disagreement, entropy, and variance perform similarly well, while self-confidence is not working because llms are prone to be over-confident. we conduct our experiments on eight datasets, spanning arithmetic reasoning, commonsense reasoning, and symbolic reasoning. experimental results demonstrate the effectiveness of our proposed method by outperforming the competitive baseline models. further analyses of different uncertainty metrics, pool sizes, zero-shot learning, and accuracy-uncertainty relationship display the benefits of each proposed module and reveal their effects. our contributions are threefold: 1) we propose to judiciously select the most helpful and informative questions for annotation, reducing the human engineering workload. 2) we introduce an effective uncertainty-based question selection strategy with several different uncertainty metrics. 3) our proposed method surpasses competitive baseline models by a large margin on multiple reasoning tasks. to the best of our knowledge, our work is the first to demonstrate the benefits of active question selection in chain-of-thought prompting for solving complex reasoning tasks. ## active-prompt the schematic illustrations of our proposed approach are illustrated in figure 1 . given l unlabeled training data d tr " tq 1 , q 2 , ..., q l u and m test data d te " tp 1 , p 2 , ..., p m u with each q and p indicating the question without any answer or reasoning steps, our goal is to annotate only n questions from d tr as few-shot exemplars by constructing a new exemplar set e " tpq 1 , c 1 , a 1 q, pq 2 , c 2 , a 2 q, ..., pq n , c n , a n qu with reasoning steps c and the answer a. then, we use e to prompt the test data d te and obtain the predictions. in this section, we explain how to select the n most uncertain questions and annotate them. ## experimental settings in this section, we describe the details of the datasets and evaluation metrics, the baseline models, and the implementation in the following three subsections. more details are included in appendix a. ## experimental results the experimental results are displayed in arithmetic reasoning: active-prompt achieves the best performance compared with all baseline models, indicating the superiority of our method. compared with the competitive baseline, self-consistency, active-prompt (d) outperforms it by an average of 2.1% with code-davinci-002. a larger improvement is observed with text-davinci-002, where active-prompt (d) improves over self-consistency by 7.2%. we notice that with code-davinci-002, the largest improvement is observed in gsm8k (4.2%) and aqua (3.1%). one possible reason is that these two datasets do not require the transferability of cot prompts because we can directly select and annotate the questions from their own training set. however, asdiv, svamp and singleeq do not have training data, so we need to transfer the annotated cot from gsm8k to them. this suggests that how to better transfer prompts from one task to another is considered an important future research direction. commonsense and symbolic reasoning: consistent improvement is observed in commonsense reasoning and symbolic reasoning tasks. active-prompt outperforms self-consistency across all three tasks. note that we test the out-of-distribution setting on letter (4), which is more challenging, and active-prompt still achieves the best performance compared with all baseline models. ## conclusion in this paper, we proposed active-prompt to elicit reasoning abilities in large language models (llms). inspired by the idea of annotating reasoning steps to obtain effective exemplars, we aim to select the most helpful questions for annotation judiciously instead of arbitrarily. for this purpose, we propose an uncertainty-based active selection strategy to determine which questions are the most important and helpful to annotate from a pool of task-specific questions. we introduce four different strategies of uncertainty estimation for active-prompt: disagreement, entropy, variance, and selfconfidence. these four strategies characterize uncertainty from different perspectives, and we primarily apply disagreement and entropy. empirically, active-prompt achieved a promising performance on eight widely used datasets for arithmetic reasoning, commonsense reasoning, and symbolic reasoning. further analyses of different uncertainty metrics, annotators, pool sizes, zero-shot learning, and an accuracy-uncertainty relationship demonstrate the effectiveness of our method.