Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning (2024)

Abstract: Large language models (LLMs) have been shown to perform better when asked to reason step-by-step before answering a question. However, it is unclear to what degree the model’s final answer is faithful to the stated reasoning steps. In this paper, we perform a causal mediation analysis on twelve LLMs to examine how intermediate reasoning steps generated by the LLM influence the final outcome and find that LLMs do not reliably use their intermediate reasoning steps when generating an answer. To address this issue, we introduce FRODO, a framework to tailor small-sized LMs to generate correct reasoning steps and robustly reason over these steps. FRODO consists of an inference module that learns to generate correct reasoning steps using an implicit causal reward function and a reasoning module that learns to faithfully reason over these intermediate inferences using a counterfactual and causal preference objective. Our experiments show that FRODO significantly outperforms four competitive baselines. Furthermore, FRODO improves the robustness and generalization ability of the reasoning LM, yielding higher performance on out-of-distribution test sets. Finally, we find that FRODO’s rationales are more faithful to its final answer predictions than standard supervised fine-tuning.

## introduction chain-of-thought (cot) reasoning techniques have been shown to improve the performance of large language models (llms) by generating step-bystep reasoning traces before generating a final answer @xcite . many works suggest that the reasoning process described in cot explanations may be a possible description of how models make predictions @xcite @xcite . however, despite the remarkable success of cot in many reasoning tasks, recent works show that llms-generated reasoning traces can be incorrect @xcite and unfaithful @xcite . reasoning implicitly involves two steps: identifying the rules and facts (inference chains) necessary to reach a conclusion and then robustly using them to reach said conclusion @xcite . our paper studies whether llms reliably use inference chains to arrive at a conclusion. 1 in standard cot, llms can generate plausible explanations with the final answer not necessarily guaranteed to follow the reasoning chain or imply a causal relation between the reasoning chain and the model's outcome @xcite . most recent efforts have either focused on the performance of llms on various reasoning tasks or their faithfulness in cot generation, ignoring the sequential relationship between cot and the final answer @xcite . in this work, we address this gap by introducing a methodology for interpreting the relationship between the cot trace and the final answer based on causal mediation analysis @xcite . causal mediation analysis is a method of causal inference that studies the change in a response variable following an intervention or treatment. more concretely, we use this method to measure and interpret the contribution of a reasoning chain (mediator) to the final answer (observed output), as shown in fig. 1 . we propose multiple interventions on the model inputs and mediators (reasoning chain) to unveil the causal effect of specific reasoning steps in a model's output. we apply this framework and study the causal impact of cot rationales on the behaviour of twelve different state-of-the-art llms on three different complex reasoning tasks (mathematical, commonsense, and causal understanding). we observe a large variation across tasks and models in how strongly reasoning traces causally affect the model's prediction. in particular, we find that instruction-tuned models (gpt-3.5-instruct, @xcite have a stronger causal effect on the final answer when conditioned on the reasoning trace than models trained with rlhf (e.g., chat-gpt; llama-2-7b-chat, @xcite . similar to @xcite , when we intervene in the reasoning problem, we observe that chatgpt and gpt-3.5-instruct are inconsistent at generating plausible reasoning chains. finally, we find @xcite only changes its answer 30% of the time when conditioned on perturbed counterfactual reasoning chains. in figure 1 , we see one example where gpt-4 does not faithfully change its final answer when provided with intervened counterfactual cot. these results indicate two issues: (i) llms can generate unfaithful and implausible reasoning chains, and (ii) llms are inconsistent when reasoning over their own generated reasoning traces. to address these issues, we introduce a novel method, frodo, comprising two modules. the first module tailors small-sized lms to generate correct reasoning chains (inference module), while the second module takes the reasoning chains as input and faithfully reasons over them to arrive at the correct answer (reasoning module). to learn to generate correct reasoning chains, we use the dpo algorithm @xcite , which enables the model to prefer correct reasoning chains over counterfactual ones with implicit feedback. instead of relying on human labeling, we obtain preference data by prompting llms to generate correct and counterfactual reasoning chains. second, we train another small-sized lm to improve the causal effect between the reasoning chain and the final answer using a counterfactual and causal preference ranking objective. we evaluate frodo on four reasoning tasks (quarel, strategyqa, openbookqa, qasc) using multiple model backbones of different scales, and demonstrate that frodo achieves an absolute accuracy improvement of 2% ∼ 3% over standard supervised fine-tuning or cot distillation methods. we assess robustness by examining how models alter their answers when intervened with counterfactual reasoning chains. frodo exhibits significant (+4.5%) improvement in robustness. finally, frodo generalizes better to out-of-distribution test sets, showing a +2.6% performance improvement over supervised fine-tuning. our code and data are publicly available foot_1 . ## frodo in this section, we introduce frodo, a framework that tailors small-sized lms (<10b parameters) to be strong rationalizers and perform reasoning faithfully over the rationales. frodo aims to improve the synergy between the reasoning chain and the final answer. we first describe how we obtain silver reasoning chains from llms ( §3.1). then, we introduce our inference module that trains a model to generate rationales ( §3.1) followed by the reasoner module and its training objectives ( §3.2). ## experiments datasets. we conduct the causal mediation analysis on three datasets: strategyqa @xcite , gsm8k @xcite , and causal understanding @xcite . we evaluate frodo on four datasets: strategyqa, quarel @xcite , openbookqa @xcite , and qasc @xcite . we report more details about each dataset in app. a.3. for all the datasets, we do not use human-written rationales. we used rationales generated by prior work @xcite using gpt-3 (text-davinci-003) as silver rationales for supervision. for counterfactual rationales, we use chain-of-thought prompts on these datasets (table. 19) and sample 2 rationales for each training instance with a temperature of 0.5. ## results in table . 2, 3, we report the results of the causal mediation analysis for twelve models. in section §4, we provide the details about the implementation, evaluation metrics and datasets. natural direct and indirect effects. we first evaluate the indirect and direct effects of the reasoning chain and reasoning problems on the final outputs. for models (>100b) with the emergent ability to generate plausible reasoning chains, we report natural indirect effects and direct effects (see §2). ## analysis generalization. the idea is to test our model's capability to determine if it can improve out-ofdistribution (ood) generalization. ## conclusion in this work, we perform a causal mediation analysis to study the indirect effect of cot on the final output of twelve llms. our experiments show large variations across tasks and models in how strongly reasoning traces causally affect the model's prediction. llms generally do not reliably use their intermediate reasoning steps when generating an answer. we introduce frodo that tailors small-sized lms to generate correct reasoning chains and faithfully reason over them to arrive at the correct answer. experiments show that our method outperforms strong baselines on four reasoning tasks, including out-of-distribution settings. rq7: what is the causal effect of open-sourced models (<100b) in natural setting? we experimented with open-sourced larger models llama-3-70b (zero-shot setting) for natural indirect effect. however, we have observed that 70b models do not have the emergent abilities to generate coherent/meaningful reasoning steps in zero-shot settings. similar to our observation, previous studies @xcite also discussed that only @xmath0.