Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness (2025)

Abstract: Chain-of-thought (CoT) prompting demonstrates varying performance under different reasoning tasks.Previous work attempts to evaluate it but falls short in providing an in-depth analysis of patterns that influence the CoT. In this paper, we study the CoT performance from the perspective of effectiveness and faithfulness. For the former, we identify key factors that influence CoT effectiveness on performance improvement, including problem difficulty, information gain, and information flow. For the latter, we interpret the unfaithful CoT issue by conducting a joint analysis of the information interaction among the question, CoT, and answer. The result demonstrates that, when the LLM predicts answers, it can recall correct information missing in the CoT from the question, leading to the problem. Finally, we propose a novel algorithm to mitigate this issue, in which we recall extra information from the question to enhance the CoT generation and evaluate CoTs based on their information gain. Extensive experiments demonstrate that our approach enhances both the faithfulness and effectiveness of CoT.

## introduction recently, with chain-of-thought (cot) techniques @xcite , large language models (llms) are able to reason on complex tasks @xcite @xcite . by scaling the cot process using reinforcement learning (rl), llms can even surpass human performance in competition-level mathematical problems @xcite . however, despite the significant success of the cot, some studies find that it demonstrates poor performance on certain tasks @xcite @xcite . in some cases, using cot for the model's reasoning is unnecessary or even harmful @xcite . these conflicting findings motivate the need for a systematic analysis of the cot. to this end, a series of studies evaluating cot's performance has commenced @xcite @xcite , which can be mainly divided into two lines: on the one hand, some works assess cot based on its effectiveness. they measure the accuracy improvements brought by the cot across different tasks and identify task types where cot is effective @xcite @xcite . on the other hand, some works evaluate the cot based on its faithfulness @xcite . they investigate the consistency between cots and final answers by analyzing the causal relevance linking them. @xcite @xcite . effectiveness is result-oriented, focusing on whether cot can enhance the quality of reasoning outcomes; whereas faithfulness is process-oriented, concerned with whether the reasoning process of cot genuinely influences the results. though these works have made great progress, they lack an in-depth analysis of the patterns influencing cot performance. for the effectiveness evaluation works, they draw conclusions like cot performs well in tasks involving mathematical symbols, but does not explore the underlying factors influencing these conclusions @xcite . for the faithfulness evaluation works, they primarily design various methods to determine whether cot is faithful, but lack an explanation for the issue of cot unfaithfulness. @xcite @xcite . in this paper, we focus on analyzing key patterns that influence the cot's performance from both effectiveness and faithfulness perspectives. from the effectiveness perspective, we identify three factors that contribute to cot's final improve-ment, including problem difficulty, information gain, and information flow. we start by splitting questions into various difficulty levels and comparing the model's accuracy on them, from which we find that cot is more effective on harder problems. then, we calculate the information gain brought by cot for questions across different tasks and demonstrate cot with more additional information tends to be more effective. lastly, we consider the internal information flow during model reasoning. through the experiment, we conclude that the more information interaction increases with the cot process, the more effective the cot becomes. from the faithfulness perspective, we discover that there exist non-negligible unfaithful cot issues in logical reasoning, where an incorrect cot can still lead to the correct answer. we further interpret this issue by jointly analyzing the information interaction among question, cot, and answer. through it, we identify three patterns that lead to the cot's unfaithfulness: (1) cot loses key information from the question; (2) cot transfers less information to the answer; (3) the model recalls correct information from the question when answering. at last, we explore the relationship between the above two perspectives. a novel algorithm called question information recall and enhancement (quire) is designed to mitigate the unfaithful cot issue. in it, we first generate a raw answer to recall correct information from the question, then use this extra information to prompt the generation of a new cot generation. finally, we employ the cot information gain as the weight to vote for the final answer. through extensive experiments, we not only demonstrate that our method can mitigate unfaithful issues, but also show that cot faithfulness is a key factor in influencing cot effectiveness. in summary, our key contributions are as follows: (1) we identify key factors that influence cot's effectiveness on different reasoning tasks, including problem difficulty, information gain, and information flow. (2) we interpret the unfaithful cot issue by jointly analyzing the information interaction among question, cot, and answer. based on experimental results, we demonstrate that the reason is that llms retrieve correct information (lost in the cot) directly from the question when predicting answers. (3) as an application of our findings, we design a new method called quire, which effectively improves the cot's performance from the effectiveness (up to 2.4% improvement) and faithfulness (up to 5.6% improvement). this indicates that enhancing cot faithfulness can lead to an improvement in cot effectiveness. our code is available at: https://github.com/bugmakerzzz/better_cot . ## what makes cot effective in this section, we investigate what factors make the cot effective in certain reasoning tasks. specifically, we start with evaluating the final accuracy improvement of cot on different tasks ( §3.1). then we study the impact of three different factors on the final performance of cot, including problem difficulty ( §3.2), cot information gain ( §3.3), and the information flow between cot and answer ( §3.4). ## what makes cot unfaithful in this section, we aim to analyze the cot from the faithfulness perspective. concretely, we first identify the unfaithfulness problem in different tasks ( §4.1). next, we analyze the issue by examining the information interaction among the three key components of reasoning (as illustrated in figure 7 ), including question and cot ( §4.2), cot and answer ( §4.3), question and answer ( §4.4). ## from unfaithful cot to effective cot since we analyze the cot from two different perspectives in the former experiments, what is the relationship between them? in this section, we demonstrate that mitigating the unfaithful issue can lead to improvements in final performance. in other words, the faithfulness of cot ( §4) is a key factor in influencing the cot effectiveness ( §3). ## conclusion in this paper, we focus on analyzing the cot performance in reasoning tasks. specifically, we identify the factors influencing cot effectiveness and interpret the mechanism behind cot unfaithfulness. for the former, we conduct extensive experiments to demonstrate that question difficulty, informa-tion gain, and information flow all contribute to cot's performance improvement. for the latter, we capture the information transfer among questions, cots, and answers in the reasoning process. the experimental results indicate that the information recall mechanism during answer predictions leads to unfaithful cot issues. at last, we design the quire method as a preliminary application of our findings, which significantly improves cot performances from both perspectives.