What Makes Chain-of-Thought Prompting Effective? A Counterfactual Study (2023)

Abstract: The effectiveness of Chain-of-thought prompting (CoT) has been widely recognized, but the underlying mechanisms behind its success, the reason why it just works for a wide range of tasks, remains an open question. To investigate this, we employ a counterfactual prompting approach, systematically manipulating elements of examples used in a few-shot prompt, and testing the consequences on model behavior. This allows us to understand the relative contributions of prompt elements such as symbols (digits, entities) and patterns (equations, sentence structure) on in-context learning. Our experiments with three different large language models (LLMs) reveal several key findings. First, the specific symbols used in the prompt do not significantly impact the model’s performance. However, consistent patterns in examples and specifying text in style frequently found on the web are crucial. Second, our findings suggest that the necessity of accurate few-shot examples depends on their role in communicating task understanding. We identify tasks where inaccurate few-shot examples hurt and, surprisingly, tasks where they improve performance. Additionally, we find that the intermediate steps in CoT may not necessarily facilitate learning how to solve a task, but instead efficiently convey task understanding (what) to the model. Furthermore, CoT leverages LLMs to fill in missing commonsense information, particularly helping difficult reasoning problems and long-tail questions.

## introduction large language models (llms) have demonstrated remarkable performance in various complex tasks using a small number of examples-a paradigm known as few-shot learning @xcite ˚equal contributions. work done when aman was a student researcher at google research, brain team (now google deepmind). 1 code and data available at https://github.com/ reasoning-machines/prompt-lib/ @xcite . this progress has been significantly boosted by chain-of-thought prompting (cot) and its variants @xcite @xcite , which have been proven to further enhance the llms' capabilities @xcite @xcite @xcite . despite its demonstrated effectiveness, the underlying mechanisms behind cot still need to be fully understood. a common explanation draws a parallel with human thinking, in which individuals often reflect on a problem before arriving at a solution @xcite . while this analogy is intuitive, it does not fully explain the reasons for cot's success, including when and how the cot mechanism operates. since llms are trained to predict the next token in a given context, there might be a more systematic explanation behind the successes and failures of cot. this study aims to explore the mechanism behind cot, providing insights into its operation. our approach involves modifying different components of the examples utilized in the few-shot prompt, and assessing the impact of these changes on the final performance (figure 1 ). specifically, we pinpoint the key elements of an example in fewshot prompting as: symbols (e.g., digits, dates) and patterns (e.g., equations, templates, sentence structure). we then apply counterfactual prompting @xcite where all components except one are held constant-for instance, replacing symbols like numbers with greek letters. the effect of each component is then assessed by comparing the performance differences between prompt variations. our experimental approach spans four diverse reasoning tasks and is implemented across three major language models-palm, gpt-3, and codex, yielding several surprising findings: ## role of patterns in this study, we define a pattern as a template that is present in all examples within a given task. the role of patterns is mainly to guide the task towards reaching its objective and hence can manifest differently depending on the task. for example, in gsm-8k patterns are structural, whereas in sports patterns represent specific rules. ## additional surface-level manipulations in addition to symbols and patterns, we delve into surface-level manipulations of text. these manipulations encompass changes to tokens that do not directly contribute to task-specific semantics but may nonetheless impact a language model's understanding and performance. in this section, we scrutinize the effects of these surface-level alterations in our prompts and examine their influence on the outcomes. text with altered grammatical style. first, we examine the impact of yodish, a syntactically valid question / thought prompt type solve rate mathematical (@xmath0.11%, @xmath1.37%) thought: shawn started with 5 toys. if he got 2 toys each from his mom and dad, then that is 4 more toys. c pat_inconsistent ppq ( thought: jamal murray and being perfect from the line are both part of basketball. c pat_inconsistent ppq ( c pat_only ppq ( c pat_wrong ppq ( question / thought solve rate mathematical (@xmath2.11%, @xmath3.37%) with 5 toys, shawn started. 2 toys each from his mom and dad, if he got, then that is 4 more toys. 5 + 4 = 9. 23.22% sports (@xmath4.08%, @xmath5.67%) a basketball player, jamal murray is. perfect from the line, is part of basketball being. 68.26% date (@xmath6.61%, @xmath7.18%) 04/19/1969, today is. later is one day after today, 24 hours, 04/20/1969, which would be. 30.75% but non-standard style of english, on the model's performance @xcite . in yodish, the xsv sentence structure is prevalent, where x is a phrase that complements the verb v, and s is the subject. for example, the sentence "bryce harper is a baseball player" would be rearranged in yodish as "a baseball player, bryce harper is". this style presents a greater challenge for the model, as it is less frequently encountered in typical training data. this makes it a valuable test case for evaluating how textual structure influences model performance. we experiment with three variations of prompts: (a) c text_yodathoughts ppq: thoughts, (b) c text_yodaquestions ppq: questions, and (c) c text_yoda ppq: both questions and thoughts in yodish. as shown in using codex as the base model, we performed additional experiments with other forms of less common patterns in standard english grammar: (a) passive voice and (b) nested clause. specifically, we modified the prompts to rewrite the original thought (e.g., shawn started with 5 toys. if he got 2 toys each from his mom and dad, then that is 4 more toys. 5 + 4 = 9.) in passive voice (5 toys were initially had by shawn. 4 more toys were received by him, 2 each from his mom and dad. 5 + 4 is 9) and using nested clauses (given that shawn had 5 toys, and considering he received 4 more from his parents, the total is 9 toys). both these variations led to a drop in performance. for passive voice the solve rates for gsm-8k, sports, and date dropped to 53.0% (-12.6%), 90.3% (-8.0%), and 65.9% (-3.3%) respec-tively. for thoughts written with nested clauses, the solve rates decreased to 55.5% (-10.1%) for gsm-8k, 90.3% (-8.0%) for sports, and 66.4% (-2.8%) for date. these results indicate that even within standard english grammar using less common variations cause discernible drops in task solve rates. shuffled and random thoughts. finally, we test the impact of altering the order or context of the text. we experiment with random thoughts (c text_rand ppq), where each thought is replaced by a semantically correct but randomly chosen thought from another example, and shuffled thoughts, where we either shuffle the words within a thought (c text_inter_shuf ppq) or across thoughts (c text_intra_shuf ppq). these experiments were designed to assess the model's dependency on the logical sequence and context of the text. the considerable decrease in performance indicates the model's reliance on coherent and contextually relevant information. the performance decrease was significant across all models, suggesting a universal dependency on coherent and contextually appropriate text among these models. in this section, we summarize our findings and present some key takeaways. takeaway i. cot helps in reinforcing task understanding. symbols and patterns can be significantly altered as long as they communicate task intent (what has to be done). in some cases (such as sorting), deviating from standard patterns may be beneficial if they more effectively communicate the task. while few-shot prompting is often called in-context learning, our findings indicate that prompts serve more as a means of reminding the model of the task that needs to be solved. takeaway ii. cot helps in eliciting commonsense knowledge. examples in a cot prompt share a key property: they help to fill in the information in the prompt. for instance, in ➥ q3 of appendix- takeaway iii. cot helps difficult examples and long-tail questions. we analyze cases where cot is exclusively correct (i.e., cot is correct and direct prompting is wrong). for gsm-8k, we find that the average number of entities in questions solved exclusively by cot is 3.98 compared to the overall average of 3.62, a statistically significant difference (difference of means t-test @xmath8.04). similarly, in the sports dataset, we find that cot is exclusively correct for rare entities: the average number of google search results for activities for which cot exclusively yields correct answers is «52ˆlower compared to the ones for which c pat_only ppq exclusively lands correct answers appendix- ccot: concise chain of thought a central observation of our study is that when few-shot examples effectively convey the task, llms can successfully harness them. moreover, a consistent pattern within the prompts that aligns with the task significantly enhances the model's ability to generate the correct response. to investigate whether a consistent pattern and the ability to fill in missing information are sufficient, we create a concise version of the chain of thought (cot) prompts, named ccot, that retain the essential information while removing unnecessary tokens. for gsm-8k, we randomly select questions from the training set whose thoughts are shorter than cot. for sports, a thought such as jamal murray is a basketball player. being perfect from the line is part of basketball was streamlined to jamal murray ñ basketball. perfect from the line ñ basketball. similarly, in today is 04/19/1969. 24 hours later is one day after today, which would be 04/20/1969 was converted to today is 04/19/1969. 24 hours (one day) later is 04/20/1969. ## related work and discussion this paper intersects with a growing body of work on prompting and large language model reasoning @xcite @xcite @xcite . role of accurate few-shot examples. @xcite find that label correctness is not crucial for the success of the models, and even random labels might lead to competitive performance. building on this work, @xcite find that the role of the correctness of the labels might be taskdependent. a concurrent body of work has also explored the reasons behind the effectiveness of chain-of-thought-prompting and shows that even wrong cot prompts can lead to strong performance @xcite . our findings complement and concur with the findings of these works and is mutually reinforcing, but go beyond the notion of the accuracy of examples. specifically, we manipulate various aspects of symbols and patterns (with correctness being one of the aspects) to examine their role in the success of cot. further, in addition to comparing the final results (outcome), we also focus on the mechanism (attention patterns). while @xcite primarily evaluates the effectiveness of cot for reasoning and question-answering tasks by introducing drastic changes to prompts to illustrate invalid reasoning, our work adopts a broader approach. we introduce counterfactual studies encompassing both subtle and significant modifications to cot. specifically, we assess cot's effectiveness under two scenarios: (1) where the reasoning flow remains but with incorrect symbols and patterns, and (2) where the reasoning flow is intentionally disrupted, such as through prompt shuffling or the introduction of random texts. in a related vein, @xcite investigates the effects of incorrect calculations and word omission/masking in cot. our work extends this by exploring the influence of out-of-distribution (ood) symbols, inconsistent patterns, exclusive use of symbols or patterns, and varied grammatical styles. few-shot learning or few-shot reminding? our results resonate with the work of reynolds and mc-donell (2021); @xcite , who found that one of the key roles played by the prompt is to remind the model of the underlying task. finally, xie et al. @xcite show that in-context learning enables a large model to infer a shared concept between the examples, possibly leading to better task understanding. our studies on the role of prompt, especially examples where wrong examples lead to better output (e.g., for sorting), provide further empirical evidence for these findings. finally, in concurrence with @xcite 's finding that pre-training term frequencies partly account for the success of few-shot methods, our experiments on sports shows that cot method is helps difficult questions involving personalities and activities less commonly found on the web. ## conclusions our study suggests that the underlying mechanisms behind the effectiveness of chain-of-thought prompting (cot) may be more complex than previously thought. we find that even large substitutions, like replacing the digits in few-shot examples with greek letters, do not affect model performance. however, simple grammar or word-order changes can have catastrophic effects. these results, along with other findings, suggest that the effectiveness of cot may stem from its ability to efficiently convey task understanding (what) to the llm. our results indicate that a combination of consistent, easy-to-mimic patterns (templates) and a strong llm that can fill missing commonsense is the recipe for effective cot. we hope to use this research to develop better prompting techniques and more robust language models for various tasks.