Paper: CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (ICLR-2024)

Reviewer comment (Novelty):
I'm not convinced the CRITIC framework is novel enough to count as a contribution. The idea idea of using natural language feedback [1, 2, 3, 4] that guides LMs in revising their responses is pretty old as is the idea of using tools [5,6]. I agree the authors provide a nice unifying framework and some new downstream tasks (e.g. toxicity with PerspectiveAPI), but these don't seem to be pass the bar for ICLR.

Author response:
Thank you for raising these concerns! Here we discuss why we believe CRITIC is novel, and more than a trivial blend of existing works on self-correction [1, 2, 3] and tool use [4,5]. While these are crucial fast-moving areas of LLMs, CRITIC offers unique findings, integrated frameworks, and valuable insights, making CRITIC a distinct contribution.

The relevant papers in these two fields you mentioned actually **have very different research perspectives and stances from CRITIC**:

### **Intrinsic Self-Correct with NL feedback**

Works include Self-Critique [6], CAI [7], Reflexion[1], Self-Refine [2], and others [8-10] you mentioned, they prompt or train language models to correct their results. In contrast, our study is the first to demonstrate that such a "Self-Verification and Self-Correction" approach has proven to be remarkably unreliable across diverse tasks and various LLMs (Sec. 4 and Appendix D.1). Specifically, **modest improvements or even deterioration are observed universally using self-correct without external feedback**. Consequently, CRITIC emphasizes the importance of feedback from external interactions for the consistent self-improvement of LLMs. The proposed framework is general and has proven effective across multiple tasks. These profound reflections on unreliable self-correction of LLMs and crucial findings on the importance of external feedback can be important learnings for the community, as recognized by Reviewer [rWYL].

### **The Unreliability of Self-correction**

Moreover, CRITIC further delves into the core reason behind the unreliability of self-verification from the perspective of uncertainty estimation, as shown in Appendix D.1. Essentially, the models are **incapable of accurately identifying "what they know" (i.e., LLMs don't know what they know)** [12] without relying on external tools. Therefore, without the aid of oracle verification (employed in many contemporary works such as Reflexion [1] and Self-Refine [2]), self-correction might surprisingly deteriorate performance for many tasks, even worsening the initial answer (as demonstrated in Table 2, 3 under CRITIC w/o Tool, and in Table 8 under Self-Refine).

As noted by Reviewer [dDfr], a nice recent follow-up work to CRITIC, titled "LLMs cannot self-correct reasoning yet" [14] extends the study of CRITIC (which cites CRITIC numerous times). It further validates and expands our findings on the unreliability of Self-Verify and Self-Correct in CRITIC using GPT-4 in more settings on many reasoning tasks.

### **Tool-Use**

Another category of related work is Tools Augmented LMs, such as ReAct [4], PoT [5], and Toolformer [17]. We would like to clarify that these works significantly differ from CRITIC as they focus on tool learning. To the best of our knowledge, **none of them consider using tool-interaction as faithful feedback to iteratively improve the model, which is the key component of CRITIC.**

In conclusion, CRITIC is the first to unveil the unreliability of self-verification and self-correction across diverse tasks and LLMs of various families and sizes. By initially highlighting the challenges LLMs encounter in self-verification [12], self-correction [6,7], our goal is to rectify any potential overestimations of these LLM's abilities within the research community [1-3]. By emphasizing that feedback from external tool interaction is crucial for consistent self-improvement of LLMs, we hope our findings provide valuable insights and encourage further exploration and enhancement of self-improving LLMs.

Reviewer comment (Experiments):
The authors don't compare with other frameworks endowing LMs with self-correction and tool use, like the ones listed above.

Author response:
In fact, CRITIC has been compared with the most advanced baselines of Self-Correct and Tool Use across multiple tasks in the original paper:

- Firstly, the work you mentioned includes methods for self-correction based on natural language feedback [1-3, 6-10], which is akin to the "CRITIC w/o Tool" baseline implemented and compared in many tasks in our paper, which relies on self-correction without using external tools.
- Moreover, in the QA task, CRITIC is even compared with a concurrent work, Self-Refine [2], as shown in Table 5. We also compared it with the Self-Correct [11] method that trains an additional corrector in the Toxicity Reduction task. We have conducted detailed comparisons and discussions on related baselines in all experiments and in Appendix C.1.
- Additionally, you mentioned the tool-use works like Toolformer [17] and many others [3,4]. We have actually compared the most advanced tool-use methods in each domain, such as the search engine-based ReAct [4] in the QA tasks; PoT [3] that used a Code Interpreter in the Math Program Synthesis tasks; Quark [18] and Self-Correct [11] that utilized Perspective API for Toxicity Reduction, etc.

We have ensured fairness in tool use and self-correct settings for all baselines across all these comparative experiments covering a wide diversity of tasks. These comprehensive comparisons across various tasks, LLMs, and model sizes collectively demonstrate the generality and effectiveness of CRITIC.

Reviewer comment (Theory):
I think the claim that tool use "mimic[s] human thinking and behavior" is overblown. Humans use think and work with tools very differently, typically not through a text-only interface.

Author response:
We sincerely appreciate your feedback regarding the wording used here!

Our intended message is that CRITIC is a framework designed, inspired by the process of human interaction with tools, such as humans compiling and running their own programs, receiving feedback, reflecting on issues, and going through the debugging process. We also agree that humans may exhibit a much richer inner monologue and behavior in using tools and reflecting, and language is indeed a limited interface through which LLM interacts with the world. In response to your feedback, we will revise this phrase to "allows for human-like verify-then-correct trajectories", and we hope this address your concerns.

In light of these clarifications and revisions, we kindly request that you consider increasing the review score.

Reviewer comment (Experiments):
How does the paper compare with other frameworks endowing LMs with self-correction and tool use?

Author response:
In fact, CRITIC has been compared with the most advanced baselines of Self-Correct and Tool Use across multiple tasks in the original paper:

- Firstly, the work you mentioned includes methods for self-correction based on natural language feedback [1-3, 6-10], which is akin to the "CRITIC w/o Tool" baseline implemented and compared in many tasks in our paper, which relies on self-correction without using external tools.
- Moreover, in the QA task, CRITIC is even compared with a concurrent work, Self-Refine [2], as shown in Table 5. We also compared it with the Self-Correct [11] method that trains an additional corrector in the Toxicity Reduction task. We have conducted detailed comparisons and discussions on related baselines in all experiments and in Appendix C.1.
- Additionally, you mentioned the tool-use works like Toolformer [17] and many others [3,4]. We have actually compared the most advanced tool-use methods in each domain, such as the search engine-based ReAct [4] in the QA tasks; PoT [3] that used a Code Interpreter in the Math Program Synthesis tasks; Quark [18] and Self-Correct [11] that utilized Perspective API for Toxicity Reduction, etc.

We have ensured fairness in tool use and self-correct settings for all baselines across all these comparative experiments covering a wide diversity of tasks. These comprehensive comparisons across various tasks, LLMs, and model sizes collectively demonstrate the generality and effectiveness of CRITIC.

Reviewer comment (Experiments):
It is unclear how important each Tool is to each task. Such analysis will provide further insight into where the improvements come from.

Author response:
> It is unclear how important each Tool is to each task. Such analysis will provide further insight into where the improvements come from.

Thank you for your suggestion! We will incorporate an analysis of the significance of different tools in various scenarios within the CRITIC paper.

To be specific, e.g., in knowledge-intensive tasks such as commonsense QA (AmbigNQ and TriviaQA) and multi-hop knowledge reasoning tasks like HotpotQA, the leading roles are played by web tools, and CRITIC primarily utilizes Wikipedia page browsing and Google snippet, as demonstrated by the numerous case studies in Appendix E.1; for mathematical program synthesis tasks, external knowledge is usually not required, and a code interpreter can serve the same function as a calculator. Therefore, in these experiments, our external feedback comes from error messages and execution results from the interpreter, as shown in the cases in Appendix E.2.

Reviewer comment (Experiments):
The authors used different sampling config for the experiments for different tasks: e.g. p=0.9 was used for section 4.3 which is different from p=0.5 in 4.1 and 4.2.

Author response:
> The authors used different sampling config for the experiments for different tasks: e.g. p=0.9 was used for section 4.3 which is different from p=0.5 in 4.1 and 4.2.

CRITIC have applied p=0.9 in the toxicity reduction task, diverging from the p=0.5 used in other tasks. This decision is based on the standard set in the RealToxicityPrompts benchmark [2], and is consistent with the methodology employed across all baseline studies [3], ensuring fair comparison.

Reviewer comment (Novelty):
I think this is a good paper. The motivation is strong: utilizing external feedback to enhance the model's ability. However, some recent studies [1] reported that large language models cannot self-correct themselves. I acknowledge that [1] did not involve external tools, which is different from CRITIC's setting and it is a paper after CRITIC which is not necessarily be included, but it would be more comprehensive to include a discussion with these new studies in such a fast-moving field.

Author response:
Sure and thanks for your nice words! There are numerous recent studies, such as [1] that you mentioned. It is actually a follow-up to CRITIC and it kindly cites the CRITIC paper numerous times. These papers further provide additional experiments on various tasks like reasoning [1],  graph coloring[2], and planning [3], further validating the findings of unreliable self-correction of LLMs from CRITIC and providing further insights [1,2,3]. Thank you for your very thoughtful suggestion, and we will include discussions on these new studies in our subsequent updates.

Reviewer comment (Reproducibility):
How much of the additional costs? Since calling external tools costs money. The authors should report the cost for each experiment.

Author response:
Nice suggestion! We’ll include a new paragraph discussing the cost of tool use in the Appendix, which is actually all free!

- For QA tasks, as mentioned in the footnote in Sec. 4.1, we build a Google Web Crawler to crawl the results of Google Search and web pages, and employ a caching mechanism for web search, storing about 9GB of search results from January to April 2023 during our experiments. This part of the code is separately open-sourced anonymously at https://anonymous.4open.science/r/llm-agent-web-tools. The results of the Search Engine in the paper are all obtained using this code. In addition, we will also open-source all caches after the anonymous review period ends, to ensure stability, fairness, and reproducibility in our results.
- For Mathematical program synthesis tasks, we use a local code interpreter, which is free of charge.
- For toxicity reduction tasks, we adopt PERSPECTIVE API [4] kindly provided by Google, which is also free.

Reviewer comment (Novelty):
In Appendix C.2, an important work active-prompt [2] should be included, which applies uncertainty estimation to chain-of-thought prompting.

Author response:
Thank you for your suggestion! 
We concur that Active-Prompt is a successful application of uncertainty estimation in the context of demonstration selection. In response to your request, we have referenced this paper in Appendix C.2 in our paper.

Reviewer comment (Novelty):
The primary concern with this work is its novelty. Several studies have previously demonstrated that external feedback can be instrumental in correcting LLM outputs. In fact, there is existing work within each domain addressed in this paper, such as Self-Correct ([1], using external APIs), Self-Ask ([2], employing a search engine), and Self-Debug ([3], via a Python interpreter). Notably, Self-Debug and Self-Ask have a striking resemblance to CRITIC but are not referenced.

Author response:
>  Several studies have previously demonstrated that external feedback can be instrumental in correcting LLM outputs. In fact, there is existing work within each domain addressed in this paper, such as Self-Correct ([1], using external APIs), Self-Ask ([2], employing a search engine), and Self-Debug ([3], via a Python interpreter). Notably, Self-Debug and Self-Ask have a striking resemblance to CRITIC but are not referenced.

Thank you for raising these concerns! We agree that there are numerous related works to CRITIC, given the importance and rapid development of LLM self-correction and tool-use topics. However, we would like to ****address the misconceptions and misunderstandings**** about CRITIC and its contributions when compared to many related works. This is because ****CRITIC's stance and findings significantly differ from these works****:

### **Intrinsic Self-Verification and Self-Correction**

These works include Self-Critique [6], CAI [7], Reflexion[1], Self-Refine [2], and others [8-10], they prompt or train language models to correct their results. In contrast, our study is the first to demonstrate that such a "Self-Verification and Self-Correction" approach has proven to be **remarkably unreliable across diverse tasks and various LLMs**. Specifically, modest or even deteriorated performance is observed universally using self-correct without external feedback in Sec. 4 and Appendix D.1. Consequently, CRITIC emphasizes the importance of feedback from external interactions for the consistent self-improvement of LLMs. The proposed framework is general and has proven effective across multiple tasks. These profound reflections on unreliable self-correction of LLMs and crucial findings on the importance of external feedback can be important learnings for the community, as recognized by Reviewer [rWYL].

- **Self-Correct** [11] uses the PERSPECTIVE API to build toxicity reduction pairs for training a corrector. Despite this, **its focus isn't on tool feedback but on creating value-improving pairs to train an additional corrector.** It doesn't use external feedback in math reasoning or constraint generation, primarily employing an intrinsic self-correction mechanism. It is mainly designed for smaller models like GPT-2, which have limited in-context learning capabilities. In contrast, our approach, CRITIC, iteratively optimizes outputs based on various external tool feedback, rather than solely relying on the model's own judgment. Additionally, CRITIC is a plug-and-play approach applicable to black-box LLMs, which further distinguishes it. We also provide an analysis and comparison to Self-Correct in our original paper's related works section and Sec 4.3's experiments.
- You also referenced **Self-Debug** [3], which notably, debuted concurrently with the first version of CRITIC. Self-Debug emphasizes code tasks, proposing self-explanation **“without any feedback on code correctness or error messages” (intrinsic feedback)**, and applying unit tests similar to LEVER [13]. In contrast, results from CRITIC show that self-correct without external feedback is unreliable, and CRITIC addresses general challenges of truthfulness, trustworthiness, and toxicity, using various tools like Google and external APIs. In light of your valuable feedback, we'll include a comparison with Self-Debug in our updated paper.

### **Tool-Use: Tool Augmented LMs**

Another category of very related work is Tool Augmented LMs, such as ReAct [4], PoT [5], and Toolformer [17]. We would like to clarify that **these works significantly differ from CRITIC as they focus on tool learning**. To the best of our knowledge, none of them consider using tool-interaction as faithful feedback to iteratively improve the LLMs, which is the key component of CRITIC.

- You also mentioned a "striking resemblance" to the **Self-Ask** paper, which to our understanding, also does not involve self-correction with external feedback. In fact, Self-Ask “uses a search engine to answer sub-questions” and does not focus on verifying and correcting answers. However, we believe this work falls within the broader scope of RAG and will be referenced in the related works section.

Reviewer comment (Presentation):
The settings that rely on an oracle are somewhat idealistic, and detract from the core message of the paper. It may be more appropriate to move these results to an appendix (as done by other works) to facilitate a clearer understanding.

Author response:
> The settings that rely on an oracle are somewhat idealistic, and detract from the core message of the paper. It may be more appropriate to move these results to an appendix (as done by other works) to facilitate a clearer understanding.

In fact, the oracle setting in CRITIC was established and included in the main text with reference to previous work (see Table 1 in [11]). It's worth mentioning that contemporaneous works to CRITIC such as Reflexion [1], Self-Refine [2], and RCI [19] *all* used the Oracle setting (i.e., trial-and-error) for their main results. Since you suggest that it may be more appropriate to move these results, we will adjust the placement of the Oracle setting in subsequent updates, hoping this will make things clearer for you.

Reviewer comment (Presentation):
The emphasis in Table 1 seems inconsistent. For instance, the AmbigNQ EM score of 50.0 is highlighted for Text-Davinci-003, but it is not the highest. Is this a bug or am I missing something?

Author response:
> The emphasis in Table 1 seems inconsistent. For instance, the AmbigNQ EM score of 50.0 is highlighted for Text-Davinci-003, but it is not the highest. Is this a bug or am I missing something?

Thank you for pointing out this nuance in Table 1. We acknowledge the discrepancy in highlighting the AmbigNQ scores, despite it not being the highest. This was indeed an oversight and not a bug. We have corrected this in an updated version. Additionally, we have meticulously reviewed all the tables to ensure no further inconsistencies are present. We appreciate your rigorous attitude and constructive feedback.

Reviewer comment (Evaluation):
Regarding the GSM task in a non-oracle setting, it appears that feedback from the interpreter is limited to syntactic correctness. Given the improvements, it suggests that many of the programs were initially syntactically wrong. Is this the case?

Author response:
> Regarding the GSM task in a non-oracle setting, it appears that feedback from the interpreter is limited to syntactic correctness. Given the improvements, it suggests that many of the programs were initially syntactically wrong. Is this the case?

Thanks for raising this issue!
The feedback from the interpreter is not solely about syntactic correctness. As mentioned in Sec. 4.2 and demonstrated in the cases in Appendix E.2, CRITIC leverages both "error messages and execution results", enabling it to fix syntax errors, correct runtime errors (e.g., timeout error in Listing 9 in Appendix E.2) and self-reflect on unreasonable reasoning steps and outputs (e.g., Listing 7 in Appendix E.2). Notably, in non-oracle settings, we do not only apply correction on answers where the interpreter fails, but by default, we allow LLMs to verify and then correct if necessary and "stop if the executed result remains unchanged for two consecutive revisions", as detailed in Sec. 4.2.

Moreover, to offer readers a more comprehensive understanding of the specific corrections made by CRITIC and the specific benefits derived from tool feedback, we carried out a manual statistical analysis of the types of corrections made by CRITIC on the GSM8k full test set (1319 samples).

Specifically, we identified four different categories of initial program errors: syntax errors, runtime errors, unreasonable outputs (such as irrational negative values), and other intrinsic reasoning errors. We calculated the accuracy of the initial PoT (Init), and CRITIC for each type of error. The settings for corrections are consistent with the non-oracle setting in the original paper, with up to four rounds of correction. The statistics are presented in the following table:

| Error Type          | Init (Count) | Init (Acc) | CRITIC (Count)| CRITIC (Acc)|   
|---------------------|-------------|----------|-------------|----------|  
| Intrinsic Error     | 281 (77.4%) | 0.0      | 206 (83.7%) | 26.7 |  
| Unreasonable Output | 61 (16.8%)  | 0.0      | 26 (10.6%)  | 57.4 |  
| Syntax Error        | 17 (4.7%)   | 0.0      | 11 (4.5%)   | 35.3 |  
| Runtime Error       | 4 (1.1%)    | 0.0      | 3  (1.2%)   | 25.0 |  
| All Init Errors     | 363         | 0.0      | 246         | 32.2 |

As shown in the above table:

- (1) The majority of error types in the initial PoT responses are intrinsic reasoning errors (77.4%), such as misunderstanding the question or omitting conditions. The initial responses also exhibit a relatively high proportion (16.8%) of unreasonable output errors, while syntax and runtime errors are less frequent but not absent (5.8%).
- (2) CRITIC has a high success rate in correcting unreasonable output and syntax errors (57.4% and 35.3% respectively). However, the correction rate for intrinsic errors, for which reliable feedback cannot be obtained, is relatively low (26.7%). Overall, CRITIC reduces errors in the initial erroneous samples by 32.2% in a non-oracle setting.

We hope our responses adequately addresses all your concerns! We greatly appreciate your time and effort in reviewing our paper, and your constructive feedback!