Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2023)

Abstract: Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). CoT explicitly encourages the LLM to generate intermediate rationales for solving a problem, by providing a series of reasoning steps in the demonstrations. Despite its success, there is still little understanding of what makes CoT prompting effective and which aspects of the demonstrated reasoning steps contribute to its performance. In this paper, we show that CoT reasoning is possible even with invalid demonstrations - prompting with invalid reasoning steps can achieve over 80-90% of the performance obtained using CoT under various metrics, while still generating coherent lines of reasoning during inference. Further experiments show that other aspects of the rationales, such as being relevant to the query and correctly ordering the reasoning steps, are much more important for effective CoT reasoning. Overall, these findings both deepen our understanding of CoT prompting, and open up new questions regarding LLMs’ capability to learn to reason in context.

## introduction large language models (llms) can perform new tasks during inference when prompted with a few demonstrations @xcite . chain-of-thought (cot) prompting @xcite can (figure 1 ) improve the ability of sufficiently large llms to do complex and multi-step reasoning. in addition to (query, answer) example-pair demonstrations, cot prompting includes a rationale (colored part in figure 1 ) for each example, i.e., a series of reasoning steps towards the answer, which encourages the llm to explicitly generate its intermediate reasoning process before predicting the final answer. despite its successes, there is little understanding of what makes cot prompting effective 1 our code and model input/output are available here. ## background & study formulation chain-of-thought (cot) prompting. different from the standard way of prompting language models where a set of (query, answer) pairs are given as demonstrations @xcite , cot prompting @xcite additionally includes a rationale (figure 1 , colored) for each example, encouraging the model to verbalize the intermediate reasoning steps for solving the task. such a technique has been shown to improve the performance of llms with sufficient scale on complex reasoning, sometimes to a large degree especially on arithmetic reasoning, multi-hop question answering, and symbolic reasoning. components of a cot rationale. we identify two distinct components of a cot rationale (examples in q: who is the grandchild of dambar shah? a: originally, leah had 32 chocolates and her sister had 42. so in total they had 32 + 42 = 74. after eating 35, they had 74 -35 = 39 pieces left in total. the answer is 39. a: dambar shah (? -1645) was the father of krishna shah. rudra shah was the child of krishna shah (? -1661). so the final answer (the name of the grandchild) is: rudra shah. ## how much does valid reasoning intuitively, one of the most important aspects of a chain-of-thought rationale would be its logically valid and sound reasoning. if we provide rationales with invalid reasoning steps in the demonstrated examples instead, we should expect the llm to fail to reason properly and gain little or even negative improvements compared with standard prompting (where no rationale is given), since we are teaching the llm to reason in the wrong way which could be even worse than not doing so at all. to test this intuition, we design an ablation study where we construct invalid reasoning steps for the demonstrated rationales, and measure its influence on model behavior. ## discussion the results from §4 and §5 open up new questions regarding learning to reason in context for llms, which we discuss next. do llms learn to reason from cot demonstrations? given the surprisingly high performance obtained by ablating the validity of reasoning for the in-context rationales, it can be concluded that what the llm learns from the demonstrations about how to reason properly is limited-rather, the llm has already gained a lot of such complex reasoning ability from pretraining (at least for tasks we experiment on), and the provided reasoning steps serve more as the role of an output format/space, that regularizes the llm to generate rationales that look step-by-step while being coherent and relevant to the query. moreover, results obtained from recent stronger models including text-davinci-003 and flan-palm (see appendix a.3) suggest that llms suffer further less from the ablations when they have more prior knowledge about the task. in particular, for flan-palm which is directly trained on both arithmetic reasoning and factual qa in cot fashion and hence has immense knowledge on these tasks @xcite , it could be seen that none of the ablations has significant impacts on its performance. on the positive side, this indicates that llms can effectively utilize their prior knowledge to solve new problems. however, from another perspective, if we view the invalid reasoning setting as a task where the goal is to generate invalid reasoning steps for the query, then the llm has basically failed to capture the task as it still tries to predict valid reasoning steps. this leads to the concern that llms may over-rely on their prior knowledge and ignore important information in the context that are presumably rare in the pretraining distribution, including those that are crucial for specifying the task semantics @xcite . can llms learn to reason in-context? we note that what we find does not in any way diminish the potential of learning to reason in context for llms; recent work has also shown evidence that learning in context is possible and could be powerful @xcite . rather, our findings show that the existing successes of cot are not sufficient for establishing that llms are good few-shot learners of reasoning; instead, the pretraining corpora have already forged them to be good reasoners on the tasks being evaluated, and the main role that the demonstrations play is to elicit such reasoning skills. reflections on benchmarking few-shot reasoning. an important topic on benchmarking in the era of large pre-trained language models is to quantify the level of prior knowledge the llm has gained about the end task being evaluated, which is crucial for assessing how well can the model truly extrapolate from pretraining and acquire new skills (chollet, 2019). one direct way is to look into the pretraining corpora when it is accessible, e.g., @xcite investigates the correlation between the model performance and the frequency of terms from the test instances in the pretraining data. however, the pretraining corpora are not always accessible, and low-level statistics are usually not adequate when the topics of interest are abstract and highlevel skills such as reasoning. along this direction, our work could be regarded as a way to approximately quantify the prior knowledge that the llm possesses on multi-step reasoning. our findings indicate that evaluations on alternative benchmarks where llms have less prior knowledge are needed to more faithfully assess the llms' abilities on learning to reason from few-shot demonstrations. ## related work there have been several subsequent work of chainof-thought prompting since its introduction. @xcite proposes to sample a diverse set of reasoning paths instead of performing greedy decoding, and marginalize over the sampled paths to select the most consistent answer. zhang et al. (2023) proposes a method for automatically constructing the in-context exemplars for cot. chen et al. (2022) explores program-based cot which can better disentangle computation from reasoning. in this paper, we are primarily focused on understanding the effectiveness of the original cot a few recent work focuses on understanding/analyzing cot prompting. @xcite investigates the importance of different components of the demonstrated cot rationales by changing them to be counterfactual. they only experiment with limited ways of changing the rationales to be wrong including using incorrect calculations (e.g., "5 + 4 = 7") or entities. for most of their settings, even though the rationales are made counterfactual, they are still correct since the query is changed accordingly (see, e.g., in general in-context learning (icl), @xcite shows that for a wide range of tasks in natural language understanding with categorical label space (classification and multi-choice), ground truth input-label mappings matter very little for end-task performance, and other aspects such as the label space, overall format and the distribution of text are the key. building on this work, yoo et al. (2022) finds that the correct input-label correspondence could have varying impacts based on the task and experimental configurations, and @xcite finds that models with larger scale can override semantic priors and learn input-label mapping in context. @xcite finds that for instruction models, the performance on natural language inference tasks has small degradations under irrelevant or misleading instructions. xie et al. (2022) provides theoretical analysis of icl by formulating it as bayesian inference. our work could be viewed as an attempt to empirically under-stand icl in sequence generation tasks requiring multi-step reasoning. ## conclusion in this paper, we aim to better understand chain-of-thought prompting through a series of ablation experiments that unveil the impact of different aspects of a cot rationale. we find that 1) the validity of reasoning in the prompting examples matters only a small portion to the performance; 2) relevance to the input query and following the order along the reasoning steps are the key to the effectiveness of cot prompting. overall, our findings deepen the understanding of cot prompting, and open up new questions/reflections regarding llms' capability of learning to reason in context.