Causal Abstraction for Chain-of-Thought Reasoning in Arithmetic Word Problems (2023)

Abstract: Recent work suggests that large language models (LLMs) achieve higher accuracy on multi-step reasoning tasks when prompted to generate intermediate reasoning steps, or a chain of thought (CoT), before their final answer. However, it is unclear how exactly CoTs improve LLMs’ accuracy, and in particular, if LLMs use their CoTs to reason to their final answers. This paper tries to answer this question with respect to arithmetic word problems, by (i) evaluating the correctness of LLMs’ CoTs, and (ii) using causal abstraction to assess if the intermediate tokens produced as part of a CoT causally impact LLMs’ final answers, in line with the reasoning described by the CoT. We find that for CoT-prompted LLMs, correct answers to arithmetic problems are highly correlated with correct CoTs, and that when LLMs produce correct CoTs, they realize to a fairly large extent the causal models suggested by their CoTs. Higher degrees of realization also seem associated with better overall accuracy on the arithmetic problems. These findings suggest that some CoT-prompted LLMs may do better on multi-step arithmetic reasoning at least partly because they use their CoTs to reason to their final answers. However, for some LLMs, other internal processes may also be involved.

## introduction while large language models (llms) achieve good zero-or few-shot performance on many natural language processing (nlp) tasks @xcite , they still seem to struggle with tasks that require multiple steps of reasoning, especially when precise computations are needed e.g. arithmetic or symbolic reasoning. intriguingly, recent work suggests that llms can do better at such tasks if they are prompted to generate a chain of thought (cot) -that is, a series of intermediate reasoning steps -before producing their final answer @xcite @xcite . it remains unclear, however, how exactly cot prompting improves the llms' performance. one natural thought is that llms use their cots to perform the intermediate computations needed to solve a multi-step reasoning problem. but this hypothesis makes several assumptions that need to be verified. first, it assumes that the cots generated are relevant to, and would correctly solve, the given questions. to evaluate this, we need to go beyond the accuracy of the llms' final answers, to also assess the quality of their cots. second, even when an llm produces text that describes some correct reasoning steps, this does not imply that the llm actually follows those steps in arriving at its answer, since the cot may not represent the llm's workings faithfully @xcite . in particular, there is no guarantee that the llm is using its generated cot tokens to work out its answer. for example, there might instead be an independent underlying process within the llm that produces both the cot tokens and the final answer, such that the cot tokens have no causal effect on the answer. thus, without evidence that the cot is causally impacting the llms' answers in the right ways, we cannot say whether the llms are reasoning using their cots, or whether they are e.g. simply predicting a plausible string of words while generating the final answer separately. this paper therefore explores the extent to which cot-prompted llms actually use their cots to reason to their final answers, in the context of multistep arithmetic problems. first, we use an original 5-point scale to evaluate the quality of llms' cots, in terms of how correctly they would solve the given questions. then, we study whether the intermediate output tokens in a cot causally impact the llm's final answer, in the ways described by the cot. to do so, we use the methods of causal abstraction, which have been used in other work to structurally analyze the inner workings of llms @xcite . in our case, we model the reasoning described in a generated cot as a causal model, with intermediate steps in the cot as variables that causally determine the llm's final answer. adapting the concepts from interventionist theories of causality @xcite , we then intervene on the variables within the cot, and compare the llms' resultant answers with the expected output from the hypothesized causal model. this allows us to assess the extent to which the llm's cot realizes the causal model, and thus the extent to which the llm is using its cot to reason to its answer. we study the performance of recently released llms on arithmetic word problems, and find that correct answers under cot prompting are highly correlated with correct cots. in addition, for questions where the llms produced a correct cot and answer, the llms appeared to realize the causal model suggested by their cots to a fairly large extent. higher degrees of realization were also associated with higher overall accuracy on the arithmetic problems, though the results also suggest that other internal processes may be important too. together, these findings provide some evidence for the causal conclusion that some cot-prompted llms do better on multi-step arithmetic reasoning tasks at least partly because they reason using their cots. if this is correct, it would suggest that cots can improve not just the performance but also the explainability of certain llms on multi-step reasoning tasks, as we may then be able to bypass their internal complexities to some extent, by using their external cots to get a handle on what they are doing. this is similar to how we can gain some understanding about how people are thinking without looking into their brains, by asking them to explain their reasoning. 1 this approach requires some care, though, as our results also suggest that the picture might be more complex for some llms. ## data we focus on one type of multi-step reasoning task that has been found to benefit from cot prompting: arithmetic word problems @xcite . this is an interesting task to study as it requires a mix of linguistic skills, which llms are trained more directly for, and arithmetic skills, which llms might find more challenging. we use the gsm8k dataset @xcite , which is widely used as a standard benchmark for performance on arithmetic reasoning. it comprises 8.5k grade school math word problems, split into a train (7.5k) and a test (1k) set. each entry in the dataset comprises a string for the math question, and a separate string for the corresponding answer, which includes multiple intermediate steps of reasoning and calculator annotations (see ## models we select three recent llms for evaluation, from three different sources, all accessible by api: ## methods we divided our experiments into three parts. ## conclusion in conclusion, we found that (i) llms achieved significantly higher accuracy on the multi-step arithmetic reasoning gsm8k test set under cot vs. non-cot prompting, (ii) correct llm answers were highly correlated with correctly formulated and executed cots, and (iii) when llms produced both correct cots and correct answers, they realized the causal models suggested by their cots to a fairly large extent. for cohere command and palm bison, greater degrees of realization were associated with higher accuracy on the gsm8k test set, suggesting that some cot-prompted llms might do better at multi-step arithmetic reasoning at least partly because they use their cots to reason to their answers. the results for openai gpt-3.5, however, suggest that for some llms, other internal processes may also be involved. because we studied only a small number of examples and interventions, future work could scale up to more examples, with more robust human evaluation methods such as inter-annotator agreement. in addition, future work could explore cases where llms produce incorrect cots, to examine whether llms nonetheless realize the causal models suggested by those cots, and whether those incorrect causal models could be salvaged by appropriate interventions. this would provide more assurance that externally generated cots faithfully represent how the llms are producing their final answers, and suggest ways to improve their performance by directly intervening on their cots. future work could also explore few-shot cot prompting approaches, which may work differently from the zero-shot paradigm, and/or explore how to extend these methods to other types of reasoning tasks, beyond arithmetic reasoning. finally, it could be useful to replicate these experiments with llms whose internal representations are accessible. this way, we can compare the results with the results that we get when we intervene within those llms, and observe how such internal interventions affect the external cots. this could help us to better understand why llms differ in the degree to which their cots realize the hypothe-sized causal models, and perhaps also shed light on interesting phenomena such as the self-correcting behavior of some llms,