Paper: Large Language Models Cannot Self-Correct Reasoning Yet (ICLR-2024)

Reviewer comment (Presentation):
It is not clear which prior papers have the various problems exposed. It would be very helpful to put them in a table. Furthermore, please provide details on where prior methods rely on oracle labels, specific problems on poorly constructed pre-prompts, specific benchmark results that are wrong. For example, I do not seem to locate which part Shinn et al. (2023) have the problems.

Author response:
Thank you for your suggestion. The table below summarizes the issues of prior methods as you suggested, and we added this table to Appendix C in our revision.

| Method                                              | Issue                                      |
|-----------------------------------------------------|--------------------------------------------|
| RCI (Kim et al., 2023); Reflexion (Shinn et al., 2023) | Use of oracle labels                        |
| Multi-Agent Debate (Du et al., 2023)                | Unfair comparison to self-consistency       |
| Self-Refine (Madaan et al., 2023)                   | Suboptimal pre-hoc prompt design            |

Regarding the use of oracle labels, in Appendix C.2 of Kim et al. (2023), they stated that “we use the correct label to decide when to stop the RCI loop.” For Shinn et al. (2023), please refer to Section 4.2 of their paper: “we use exact match answer grading using the environment to give a binary success signal to the agent,” which is also revealed in their official implementation.

Reviewer comment (Novelty):
The paper exposes an intriguing problem in prior work on self-correction of LLMs that shows, to the contrary, that LLMs can not self-correct reasoning yet. However, it is still not clear how we should think about all the techniques to improve reasoning without external feedback. Does breakdown a problem into sub-problem and do step-wise verification paired with self-consistency considered self-correction? It would be very helpful to put all these techniques into perspective.

Author response:
Thank you for your suggestion. As discussed in our Section 5 “Employing self-consistency as a method of self-verification”, breaking down a problem into sub-problems and conducting step-wise verification paired with self-consistency can be helpful.

Another promising future direction is exploring how to enable LLMs to interact with and learn from external feedback (e.g., interacting with the environment), though this falls outside the scope of "intrinsic self-correction" discussed in our paper.

We thank you for recognizing the contribution of our paper to the field. Your feedback will significantly enhance the clarity and comprehensiveness of our research.

Reviewer comment (Theory):
I find the explanation in section 3.2.1—why post-hoc prompting can lead the model to go from a correct to an incorrect answer—unsatisfying. We know that the feedback prompt is changing the model output somehow. The question is *why?* I suggest providing more intuition here.

Author response:
Thank you for your feedback. The fundamental issue is that LLMs cannot properly judge the correctness of their reasoning. So the feedback prompt introduces a new context that may bias the model away from its original correct answer. We have added the explanation in the revision.

Reviewer comment (Experiments):
Have you tried combining Multi-agent debating with self-consistency?

Author response:
Yes. When we run the multi-agent debate method, we take the majority vote of the final responses of all three agents (following the setting of the original paper). We also try taking the majority vote of all responses (including the previous responses) of the agents, but do not observe performance improvement.

Reviewer comment (Experiments):
How much effort did you invest into finding the self-correct prompt you used "Review your previous answer and find problems with your answer"? Can't different variations of this same prompt lead to different results?

Author response:
The prompt "Review your previous answer and find problems with your answer" originates from Kim et al., (2023). We adhered to their prompt to replicate their experiments. We have also made efforts in designing different prompts for testing:
- In Appendix B.1 of the initial draft, we test different feedback prompts, such as “Verify whether your answer is correct, and provide an explanation.”
- In Appendix B.2 of the revised draft, we add a prompt suggested by other reviewers.
- The prompts used in previous works (Kim et al., 2023; Shinn et al., 2023; Du et al., 2023; Madaan et al., 2023) are different.We adhere to the prompts from the source papers to replicate their experiments. So our prompts in these experiments are also different.

All the results consistently show that the models’ performance on reasoning does not improve and even decreases after self-correction.

Reviewer comment (Experiments):
The second major problem is the work’s heavy reliance on a single, loaded prompt: “Review your previous answer and find problems with your answer.” This is a highly leading prompt that essentially tells the LLM that problems exist, often causing it to find issues that are not present. The paper itself acknowledges that “careful consideration of prompt design is essential.”

Author response:
A primary question raised by the reviewer concerns the reliance on a single, loaded prompt. However, this is not true. **In our paper, both in the main text and Appendix B, we have tested multiple prompts**.

First, as this paper functions partially as a critique paper, we initially follow the prompts and models used in prior studies, specifically testing three different sets of prompts from Kim et al., 2023, Shinn et al., 2023, and Du et al., 2023.

**Furthermore, we also test unbiased self-correction prompts in Appendix B.1 of the initial version of our paper, such as 'Verify whether your answer is correct, and provide an explanation.' In Appendix B.2 of the revised manuscript, we conduct additional tests on GPT-4-Turbo and Llama-2-70b using the prompt suggested by the reviewer: 'Assume that this answer could be either correct or incorrect. Review the answer carefully and report any serious problems you find.' The results consistently show that the models’ performance decreases after self-correction.** We welcome the reviewer to check the results.

Reviewer comment (Experiments):
The study omits important factors routinely used in prompt engineering that would be necessary for a proper examination of LLM self-correction. These include: focusing on GPT-4 due to its significantly greater capabilities compared to GPT-3.5 (as shown in Table 3); including diagrams like those in Figure 1 for GPT-4 and not just GPT-3.5; evaluating a set of reasonable, unbiased self-correction prompts (e.g., “Assume that this answer could be either correct or incorrect. Review the answer carefully and report any serious problems you find”); and focusing on the zero-temperature setting, which is less prone to hallucination.

Author response:
As our paper partially serves as a critique paper, our choice of temperature setting adheres to previous works. We conducted additional experiments detailed in Appendix B.2, employing the settings suggested by the reviewer (e.g., zero-temperature, the prompt suggested by the reviewer). We added the results and full error analysis in Appendix B.2 of our revision.

|                          |                      | # calls | GSM8K | CommonSenseQA |
|--------------------------|----------------------|-------|-------|---------------|
| gpt-4-1106-preview       | Standard Prompting   | 1     | 91.5  | 84.0          |
|                          | Self-Correct (round 1) | 3   | 88.0  | 81.5          |
|                          | Self-Correct (round 2) | 5   | 90.0  | 83.0          |
| Llama-2-70b-chat-hf      | Standard Prompting   | 1     | 62.0  | 64.0          |
|                          | Self-Correct (round 1) | 3   | 43.5  | 37.5          |
|                          | Self-Correct (round 2) | 5   | 36.5  | 36.5          |

>**4. “Focus on GPT-4. Include diagrams like those in Figure 1 for GPT-4, not just for GPT-3.5.”**

There are two primary reasons we test on GPT-3.5 (while also providing results for GPT-4):

(1) Alignment with previous studies: As our paper partially serves as a critique of the studies by Kim et al., 2023; Shinn et al., 2023; and Du et al., 2023, which focus on GPT-3.5, it is imperative that we follow their settings to accurately reproduce their results.

(2) Cost considerations: The cost associated with GPT-4 is substantial. For instance, conducting a single full test on GSM8K with self-correction using GPT-4 incurs a cost of approximately $200 (in total, it will cost thousands of dollars). To manage these costs, we opt to randomly sample 200 examples for our tests in our initial draft.

We appreciate your suggestion and have included diagrams for GPT-4 in both Figure 1 and Appendix B.2.

Reviewer comment (Experiments):
The statement “in the reasoning tasks studied in this paper, we did not observe any improvement through self-correction” is accurate regarding the reported experiments, but the study was not extensive enough to offer meaningful insight into the broader question of how reliably LLMs can self-correct.

Author response:
>**5. Our study focuses on the question "Can LLMs self-correct *reasoning* intrinsically?" by examining various prompts and previous studies, not on the general question of how reliably LLMs can self-correct for all tasks:**

As discussed above, in Appendix B, we have tested with different (unbiased) self-correction prompts. We have also tested three different sets of prompts from Kim et al., 2023; Shinn et al., 2023; and Du et al., 2023. All the results consistently show that the models’ performance on reasoning does not improve and even decreases after self-correction, supporting the statement that LLMs struggle to self-correct reasoning intrinsically.

Moreover, as mentioned in the title and our response 1 above, the focus of our paper is on self-correction for *reasoning*, not on general self-correction for all tasks. While we do discuss self-correction for other tasks a bit, our aim is to provide readers with a more balanced and comprehensive understanding of self-correction. Additionally, we are open to conducting any further experiments (considering the cost and time) that the reviewers might deem necessary before the revision deadline.

Reviewer comment (Evaluation):
Post-rebuttal, while additional experiments using GPT-4 with an unbiased feedback prompt and zero temperature provided interesting results on two datasets (raising the reviewer’s contribution rating from 2 to 3), these limited results are still insufficient to support the broad title claim that “Large Language Models Cannot Self-Correct Reasoning Yet.” Even with the restriction to “reasoning” tasks, the title overstates the findings.

Author response:
Thank you for reviewing our response. We would like to follow up and further address your concerns.

First, we present additional results for GPT-4-Turbo using different feedback prompts. These results are summarized in the table below, consistently demonstrating a decline in performance after self-correction. We have included these results in Table 8 of Appendix B.2.

|                               | # calls | GSM8K | CommonSenseQA |
|-------------------------------|---------|-------|---------------|
| _Feedback Prompt_: Assume that this answer could be either correct or incorrect. Review the answer carefully and report any serious problems you find. |         |       |               |
| Standard Prompting            | 1       | 91.5  | 84.0          |
| Self-Correct (round 1)        | 3       | 88.0  | 81.5          |
| Self-Correct (round 2)        | 5       | 90.0  | 83.0          |
|                                   |         |       |               |
| _Feedback Prompt_: Review your previous answer and determine whether it's correct. If wrong, find the problems with your answer. |         |       |               |
| Standard Prompting            | 1       | 91.5  | 84.0          |
| Self-Correct (round 1)        | 3       | 90.0  | 74.5          |
| Self-Correct (round 2)        | 5       | 90.0  | 81.0          |
|                                   |         |       |               |
| _Feedback Prompt_: Verify whether your answer is correct, and provide an explanation. |         |       |               |
| Standard Prompting            | 1       | 91.5  | 84.0          |
| Self-Correct (round 1)        | 3       | 91.0  | 81.5          |
| Self-Correct (round 2)        | 5       | 91.0  | 83.5          |


Regarding the feedback prompts from previous studies: In Section 3.1.1 Prompts, we state, "we mostly adhere to the prompts from the source papers. For GSM8K and CommonSenseQA, we integrate format instructions into the prompts of Kim et al. (2023) to facilitate a more precise automatic evaluation (detailed prompts can be found in Appendix A). For HotpotQA, we use the same prompt as Shinn et al. (2023)." In Section 3.3, we note, "For an unbiased implementation, we use the exact same prompt as Du et al. (2023) and replicate their experiment with the gpt-3.5-turbo-0301 model, incorporating 3 agents and 2 rounds of debate." The reviewer may refer to the previous papers or their implementations for the prompts, which are identical to those used in our experiments.

Regarding the title, we thank you for your suggestion. However, we do not fully understand the argument that "The authors argue that the title’s claim is restricted to “reasoning” tasks, but this is not much of a restriction at all." We would appreciate it if the reviewer could further clarify. If you feel that this is the main weakness, we will change the title to "Reexamining the Ability of Large Language Models to Self-Correct Reasoning" in the final version. Additionally, we have softened the statements in the paper, for example, changing 'cannot' to 'struggle to'.

We hope the additional results and the responses above address your concerns. Please do not hesitate to contact us for any further questions or clarifications. Additionally, we are open to conducting any further experiments (considering the cost and time) that the reviewers might deem necessary before the revision deadline.

Reviewer comment (Experiments):
The other feedback prompts used in the experiments (Appendix B.1) apply only to GPT-3.5, which is known to be poor at self-critique. Although the authors suggest that prior works used different prompts, Appendix A shows only the single, loaded prompt “Review your previous answer and find problems with your answer.” This further weakens the generality of the results.

Author response:
A primary question raised by the reviewer concerns the reliance on a single, loaded prompt. However, this is not true. **In our paper, both in the main text and Appendix B, we have tested multiple prompts**.

First, as this paper functions partially as a critique paper, we initially follow the prompts and models used in prior studies, specifically testing three different sets of prompts from Kim et al., 2023, Shinn et al., 2023, and Du et al., 2023.

**Furthermore, we also test unbiased self-correction prompts in Appendix B.1 of the initial version of our paper, such as 'Verify whether your answer is correct, and provide an explanation.' In Appendix B.2 of the revised manuscript, we conduct additional tests on GPT-4-Turbo and Llama-2-70b using the prompt suggested by the reviewer: 'Assume that this answer could be either correct or incorrect. Review the answer carefully and report any serious problems you find.' The results consistently show that the models’ performance decreases after self-correction.** We welcome the reviewer to check the results.

Reviewer comment (Evaluation):
For these reasons, the reviewer considers the work fundamentally unsound, as the limited findings do not justify the strong and attention-grabbing title of the paper.

Author response:
Thank you for your feedback. Based on the ICLR policy, we are not able to change the paper title during the rebuttal phase, but we will consider your suggestion in the final version. Nevertheless, we wish to clarify that our title does not mean self-correction does not work for all tasks; rather, as our original title includes the word “reasoning”, our work focuses on self-correction for reasoning. The title reflects our finding that current LLMs cannot yet enhance reasoning performance through self-correction. We have also adjusted our claims in the paper to make them more specific to reasoning.

Reviewer comment (Novelty):
Additional Post-rebuttal — The argument that restricting the title’s claim to “reasoning” tasks is a significant limitation is unconvincing because LLMs are strong at memorization but weak in many types of reasoning. Since self-critique is especially relevant to reasoning problems, restricting the scope to reasoning is not much of a restriction.

Author response:
Regarding the title, we thank you for your suggestion. However, we do not fully understand the argument that "The authors argue that the title’s claim is restricted to “reasoning” tasks, but this is not much of a restriction at all." We would appreciate it if the reviewer could further clarify. If you feel that this is the main weakness, we will change the title to "Reexamining the Ability of Large Language Models to Self-Correct Reasoning" in the final version. Additionally, we have softened the statements in the paper, for example, changing 'cannot' to 'struggle to'.

Reviewer comment (Presentation):
Upon the authors’ agreement to change the title to “Reexamining the Ability of Large Language Models to Self-Correct Reasoning,” which is aligned with the experiments, the reviewer raises the paper rating from 3 to 6.

Author response:
Regarding the title, we thank you for your suggestion. However, we do not fully understand the argument that "The authors argue that the title’s claim is restricted to “reasoning” tasks, but this is not much of a restriction at all." We would appreciate it if the reviewer could further clarify. If you feel that this is the main weakness, we will change the title to "Reexamining the Ability of Large Language Models to Self-Correct Reasoning" in the final version. Additionally, we have softened the statements in the paper, for example, changing 'cannot' to 'struggle to'.

Reviewer comment (Theory):
Question — In Section 3.2, the statement “If the model is well-aligned and paired with a thoughtfully designed initial prompt, the initial response should already be optimal” is presented as an “Intuitive Explanation.” The reviewer asks why this should be considered intuitive or expected, noting that by similar reasoning, the value of chain-of-thought prompting would itself be unintuitive.

Author response:
>**6. Questions on Section 3.2:**

The reviewer appears to have truncated our statement into an incomplete sentence. The complete sentence is, “If the model is well-aligned and paired with a thoughtfully designed initial prompt, the initial response should already be optimal *relative to the prompt and the specific decoding algorithm*.” The critical missing part, 'the response is optimal relative to the given prompt and the specific decoding algorithm', significantly alters the meaning when omitted.

The effectiveness of 'Chain-of-Thought Prompting' lies in its provision of better guidance for the model to solve the problem. Therefore, the model's response is optimal to the improved prompt (w/ CoT); it surpasses the response generated without CoT, which is optimal to a less effective prompt (w/o CoT). This contrasts significantly with the intrinsic self-correction we discuss here, where the feedback prompt (without usage of oracle labels) does not provide any helpful benefits for reasoning and may even bias the model to generate a worse response.

We thank you for your constructive critique and guidance. The suggested modifications certainly enhance our paper.

Reviewer comment (Experiments):
Only a small set of questions (200) is used on GPT4, the remaining ones apply only to ChatGPT.

Author response:
There are two primary reasons we test on GPT-3.5 (while also providing results for GPT-4):

(1) Alignment with previous studies: As our paper partially serves as a critique of the studies by Kim et al., 2023; Shinn et al., 2023; and Du et al., 2023, which focus on GPT-3.5, it is imperative that we follow their settings (e.g., testing on GPT-3.5) to accurately reproduce their results on the well-established benchmarks used in their papers.

(2) Cost considerations: The cost associated with GPT-4 is substantial. For instance, conducting a single full test on GSM8K with self-correction using GPT-4 incurs a cost of approximately $200 (in total, it will cost thousands of dollars). To manage the costs, we opt to randomly sample 200 examples for our tests on GPT-4.

A point to mention is that we do consider running tests on the full test set when the cost is reasonable. For instance, Du et al. (2023) tested only 100 examples on GSM8K with GPT-3.5, whereas we chose to run the full test set to make the results more reliable.

In Appendix B.2 of our revision, we conducted additional experiments on more examples with GPT-4-Turbo and Llama-2, employing the settings suggested by other reviewers.

|                          |                      | # calls | GSM8K | CommonSenseQA |
|--------------------------|----------------------|-------|-------|---------------|
| gpt-4-1106-preview       | Standard Prompting   | 1     | 91.5  | 84.0          |
|                          | Self-Correct (round 1) | 3   | 88.0  | 81.5          |
|                          | Self-Correct (round 2) | 5   | 90.0  | 83.0          |
| Llama-2-70b-chat-hf      | Standard Prompting   | 1     | 62.0  | 64.0          |
|                          | Self-Correct (round 1) | 3   | 43.5  | 37.5          |
|                          | Self-Correct (round 2) | 5   | 36.5  | 36.5          |

Reviewer comment (Writing):
The wording is unfortunately often trendy rather than clear: From the conclusion: "while LLMs represent a groundbreaking step forward in the realm of AI and language generation, their self-correction capabilities, particularly in reasoning, are still nascent". "Nascent" can mean anything from the self-correction capabilities being "not present", "weak" "promising" etc. More specific wording should be used.

Author response:
We appreciate your suggestion on the wording. The term “nascent” was intended to convey that LLMs' self-correction abilities, particularly in reasoning, are not yet fully effective. We have updated the conclusion section and incorporated more specific wording in the revision.

Reviewer comment (Writing):
The paper is confusing to read due to somewhat contradictory statements: In the beginning the authors emphasize that they want to investigate *intrinsic* self-correction abilities. Then, on page 3, they state: "With this in mind, we center our investigation on a pivotal query: Can large language models self-correct their reasoning?" which does not necessarily imply intrinsic self-correction, but any type of self-correction. This unclarity seems to pervade the remainder of the paper.

Author response:
Thank you for your feedback. Note that at the end of Section 2 (page 2), we specified, “For brevity, unless explicitly stated otherwise (e.g., self-correction with oracle feedback), all references to ‘self-correction’ in the remainder of this paper pertain to intrinsic self-correction.” Thus, the question “Can large language models self-correct their reasoning?” on page 3 refers to intrinsic self-correction.

Reviewer comment (Novelty):
In section 7 the authors state: "Some existing literature may inadvertently contribute to this confusion, either by relegating crucial details about label usage to less prominent sections or by failing to clarify that their designed self-correction strategies actually incorporate external feedback. Our intention in this paper is to amplify these concerns and offer a comprehensive overview of the state of “self-correction” in LLMs". This implies that the current paper was more like a survey, rather than establishing new results.

Author response:
We would like to clarify that our paper critiques and analyzes LLMs’ ability to self-correct reasoning. We present new results showing that LLM self-correction without oracle labels underperforms baselines without self-correction:
- Critique: We reexamine previous works and find that 1) previous works use oracle labels to perform self-correction, which is not practical; 2) multi-agent debate is not superior to self-consistency; 3) there is an issue with employing a well-crafted post-hoc prompt to guide the model in “self-correcting” a response generated through a poorly constructed pre-hoc prompt. 
- Analysis: We demonstrate that current LLMs struggle to self-correct reasoning intrinsically through comprehensive analysis and offer insights and suggestions for future work.

Reviewer comment (Presentation):
The title suggests that self-correction does not work, although the paper does not really argue this. In section 7 the authors remark "The title, 'Large Language Models Cannot Self-Correct Reasoning Yet', is not an outright dismissal of self-correction techniques". Generally, it would be good to use a title that is completely representative of the paper (perhaps add "intrinsic" to the title).

Author response:
Thank you for your feedback. Based on the ICLR policy, we are not able to change the title during the rebuttal phase, but we will consider your suggestion in the final version. Nevertheless, we wish to clarify that our title doesn’t mean self-correction does not work for all tasks; rather, our work focuses on self-correction for reasoning. The title reflects our finding that current LLMs cannot yet enhance reasoning performance through intrinsic self-correction.

Reviewer comment (Evaluation):
"Per the discussions in Section 3.1.3, since the idea that LLMs can self-correct their reasoning is not supported by the evidence so far, we turn our focus to the results in the intrinsic self-correction." I wasn't quite able to follow why section 3.1.3 shows this; Table 1 actually seems to show that self-reasoning is well-supported by evidence.

Author response:
As explained above, the results in Table 1 use oracle labels to guide self-correction, which is impractical and differs from the intrinsic self-correction setup. Therefore, they cannot support the claim that LLMs can self-correct reasoning (intrinsically). We have adjusted the writing to enhance clarity.