Paper: Teaching Large Language Models to Self-Debug (ICLR-2024)

Reviewer comment (Experiments):
The paper has a significant flaw due to an unfair comparison with existing techniques and the baseline, caused by different access to test input/output pairs to decide correctness of code; as a result, the reported improvement numbers over the baseline or existing work are meaningless.

Author response:
Thanks for raising the concern. We appreciate that you consider our work as innovative and well-written. We would like to clarify our experimental setup as follows.

### Self-debugging does not have different test input/output pairs to decide the code correctness compared to baselines ###

We want to emphasize that self-debugging does not utilize extra input-output pairs compared to the baselines. Specifically, for text-to-SQL generation, there is no unit test for both self-debugging and all other baselines. Therefore, the LLM needs to infer the code correctness completely by itself. As described in Section 4.3, on MBPP, we follow the baseline approaches to include 1 unit test in the problem description, and other hidden unit tests are only used for the final evaluation on the code correctness. Therefore, the LLM also needs to infer the code correctness, since passing the given unit test does not necessarily mean that the predicted code is fully accurate and can pass the hidden unit tests. Note that during the self-debugging process, the hidden unit tests are never used, and the generated code is only executed on the 1 given unit test that already shows up in the problem description for generating the initial code. For code translation, we consider the setting where all unit tests are available, since the ground truth execution results can already be obtained by executing the code in the source programming language, which also follows the evaluation setup in prior work.

### We compared to baselines that utilize code execution ###
As described in Section 2 and Section 5 (page 5), we compared self-debugging to baselines that generate multiple programs, and use code execution to select the final response.

On the Spider benchmark for text-to-SQL generation, since the unit tests are not available, the baseline takes the majority vote of execution results as the final prediction. As shown in Figure 4 (a), self-debugging from greedy decoding matches the performance of the execution-based baseline that uses 16 samples, and self-debugging consistently improves the sample efficiency. For Transcoder, our baseline utilizes all unit tests to determine the code correctness. Note that the performance of this baseline is equivalent to the reviewer’s second suggested solution, since we consider the prediction from this baseline to be accurate if any one of the generated samples is correct. Again, we show that self-debugging from greedy decoding matches the baseline that generates 10 samples, and we presented the full analysis in Appendix E.1.

### The usage of richer execution information is a strength of our self-debugging design ###

First, we would like to note that the reranking techniques LEVER and MBR-Exec also utilize code execution to select the response. The main difference between our work and these execution-based reranking techniques lies in the way we utilize the execution results. Specifically, the baseline approaches generate multiple samples all at once, thus the model can generate the same wrong code multiple times despite observing the execution errors. On the other hand, self-debugging can leverage past predictions and code execution to reduce the redundant cost of resampling wrong predictions, which is an advantage over existing reranking techniques.

Furthermore, the key motivation of our self-debugging design is to provide complementary performance gain on top of all kinds of existing code generation methods. As shown in Figure 1 and described in Section 3, the first step of each self-debugging turn is code generation, which can be implemented as any existing baseline such as greedy decoding and execution-based reranking. Therefore, the subsequent debugging step leverages richer execution information than the first step of code generation by design. We demonstrate in Table 1, Figure 4 (a) and Appendix E.1 that self-debugging does not only improve over greedy decoding, but also consistently improves the performance when the baseline utilizes multiple samples.

We hope the above discussion addresses your concerns. Please do not hesitate to contact us for any further questions or clarifications.

Reviewer comment (Evaluation):
One proposed solution is that, in self-repair, instead of using test I/O to decide if the code should enter the self-repair process, the authors could let the model itself decide correctness using only unit tests (for MBPP and TransCoder) and the task prompt. While the model may misclassify some correct programs as incorrect, it would not benefit from the external test oracle, ensuring fair comparability with existing approaches and the baseline no-repair model since all approaches would have the same access to information during evaluation.

Author response:
Thanks for acknowledging our response and being willing to update your score! We are glad that our response addressed your concerns, and we answer your followup questions below.

1. For text-to-SQL generation, you are correct that self-debugging only draws 2 samples (1 greedy + 1 self debug output), while outperforming the majority voting based on 16 solutions. Also, LLM indeed might wrongly infer the solution correctness and turn correct solutions into wrong ones. Below are the statistics using Codex:
* LLM wrongly inferred correct initial correct solutions as incorrect: 3.3%
* LLM wrongly changed initial correct predictions into incorrect ones: 1.0%
* LLM fixed initial wrong predictions into correct ones: 4.3%

We observe that even if the LLM does not have 100% accuracy of deciding whether a solution is correct, the majority of modified solutions upon the initial correct solutions is still accurate, thus most incorrect predictions of the code correctness do not directly lead to performance degradation.

2. Thanks for suggesting the enhanced baseline! You are right that the baseline in Table 2 does not have a chance to generate another solution when the greedy decoding produces the wrong code. We evaluated the enhanced baseline using Codex, and the accuracies are as follows:
* Transcoder: greedy decoding = 80.4%, enhanced baseline = 84.5%, self-debugging = 92.5%
* MBPP: greedy decoding = 61.4%, enhanced baseline = 65.4%, self-debugging = 70.8%

These results show that self-debugging outperforms the enhanced baseline, which highlights the effect of self-debugging feedback.

Note that the design of the baseline with multiple samples in Table 1 in the main paper and Figure 5 (b) in Appendix E.1 is equivalent to the enhanced baseline, except that these results were obtained with more than two trials. For example, Figure 5 (b) shows that when the baseline uses 5 samples (equivalent to the enhanced baseline with 5 trials), the accuracy is 90.9%, which is still worse than self-debugging that reaches 92.5% accuracy.

We hope these results address your questions. Please let us know your thoughts, and we are more than happy to answer any further questions.

Reviewer comment (Experiments):
The reviewer requests that the authors clearly address their concern regarding the "comparison with baseline."

Author response:
Thanks for raising the concern. We appreciate that you consider our work as innovative and well-written. We would like to clarify our experimental setup as follows.

### Self-debugging does not have different test input/output pairs to decide the code correctness compared to baselines ###

We want to emphasize that self-debugging does not utilize extra input-output pairs compared to the baselines. Specifically, for text-to-SQL generation, there is no unit test for both self-debugging and all other baselines. Therefore, the LLM needs to infer the code correctness completely by itself. As described in Section 4.3, on MBPP, we follow the baseline approaches to include 1 unit test in the problem description, and other hidden unit tests are only used for the final evaluation on the code correctness. Therefore, the LLM also needs to infer the code correctness, since passing the given unit test does not necessarily mean that the predicted code is fully accurate and can pass the hidden unit tests. Note that during the self-debugging process, the hidden unit tests are never used, and the generated code is only executed on the 1 given unit test that already shows up in the problem description for generating the initial code. For code translation, we consider the setting where all unit tests are available, since the ground truth execution results can already be obtained by executing the code in the source programming language, which also follows the evaluation setup in prior work.

### We compared to baselines that utilize code execution ###
As described in Section 2 and Section 5 (page 5), we compared self-debugging to baselines that generate multiple programs, and use code execution to select the final response.

On the Spider benchmark for text-to-SQL generation, since the unit tests are not available, the baseline takes the majority vote of execution results as the final prediction. As shown in Figure 4 (a), self-debugging from greedy decoding matches the performance of the execution-based baseline that uses 16 samples, and self-debugging consistently improves the sample efficiency. For Transcoder, our baseline utilizes all unit tests to determine the code correctness. Note that the performance of this baseline is equivalent to the reviewer’s second suggested solution, since we consider the prediction from this baseline to be accurate if any one of the generated samples is correct. Again, we show that self-debugging from greedy decoding matches the baseline that generates 10 samples, and we presented the full analysis in Appendix E.1.

### The usage of richer execution information is a strength of our self-debugging design ###

First, we would like to note that the reranking techniques LEVER and MBR-Exec also utilize code execution to select the response. The main difference between our work and these execution-based reranking techniques lies in the way we utilize the execution results. Specifically, the baseline approaches generate multiple samples all at once, thus the model can generate the same wrong code multiple times despite observing the execution errors. On the other hand, self-debugging can leverage past predictions and code execution to reduce the redundant cost of resampling wrong predictions, which is an advantage over existing reranking techniques.

Furthermore, the key motivation of our self-debugging design is to provide complementary performance gain on top of all kinds of existing code generation methods. As shown in Figure 1 and described in Section 3, the first step of each self-debugging turn is code generation, which can be implemented as any existing baseline such as greedy decoding and execution-based reranking. Therefore, the subsequent debugging step leverages richer execution information than the first step of code generation by design. We demonstrate in Table 1, Figure 4 (a) and Appendix E.1 that self-debugging does not only improve over greedy decoding, but also consistently improves the performance when the baseline utilizes multiple samples.

We hope the above discussion addresses your concerns. Please do not hesitate to contact us for any further questions or clarifications.

Reviewer comment (Presentation):
Although the introduction of this paper is clear, the methodology part is not the case. There are many components in the proposed approach: code execution, code explanation, inferring code correctness, etc. Figure 1 is helpful but still not clear enough. It would be better if there is a diagram with concrete example in the main text.

Author response:
Thanks for the feedback. We have added Figure 2 and 3 in the main paper to show the self-debugging workflow, which are abbreviated versions of Figures 6 and 7 in Appendix F. The 2 figures demonstrate concrete running examples of how self-debugging works with different prompt formats.

Reviewer comment (Evaluation):
Although the paper claims that they improve sample efficiency, I am still doubtful about this as the self-debugging approach does require generating much more tokens for debugging. I think comparing different approaches by the average the number of tokens generated for solving each problem is more fair.

Author response:
Compared to the sampling cost for obtaining the initial code for debugging, self-debugging requires the LLM to generate more tokens by design. For example, in Table 2, self-debugging generates more samples than the greedy decoding baseline. This extra cost is also needed for other related work, e.g., self-refine and Reflexion.

On the other hand, as discussed in Section 5.2.1, we demonstrate that to improve the accuracy over greedy decoding, self-debugging achieves better performance with a lower inference cost. To compare the cost in terms of the number of tokens, we consider two settings: (1) when unit tests are not available; e.g., on Spider for text-to-SQL generation; and (2) when unit tests are available; i.e., on Transcoder and MBPP for Python code generation.

On the Spider benchmark, with explanation feedback, on average a full debugging turn generates 5.5 times more tokens than simply generating the SQL query. By comparing the performance of self-debugging starting from greedy decoding (80.8% accuracy) to the baseline that samples 16 programs without self-debugging (80.7% accuracy), we can see that self-debugging also reduces the number of generated tokens compared to the baseline.

On Transcoder and MBPP, due to the availability of unit tests, the improvement with explanation feedback is generally less significant than on Spider. In this case, the unit test feedback is a better option to reduce the sampling budget. On average, a full debugging turn with the unit test feedback generates 2.5 times more tokens than simply generating the initial code, thus self-debugging is again more sample-efficient in terms of the token cost.

Furthermore, despite the fact that generating explanation and trace feedback increases the number of generated tokens, we note that one benefit of such feedback is to provide more interpretable rationales on how LLMs infer the code correctness, and how the fixes are conducted. Such natural language feedback helps us to investigate the models’ weaknesses, and potentially can be used to finetune the model and improve its overall debugging and coding ability.

Reviewer comment (Novelty):
How do you compare your approach to Reflexion [1]?

Author response:
The Reflexion paper evaluates their approach on AlfWorld and HotPotQA, which are decision-making tasks and question-answering tasks. For both benchmarks, they assume that the oracle reward is available, which tells the LLM whether the current state is successful or the current answer is correct.

On the contrary, our work focuses on code generation tasks, where the LLM does not always have access to the oracle reward about the code correctness. For example, on text-to-SQL generation where the unit tests are not available, the LLM needs to determine completely by itself when the debugging process should terminate. Furthermore, we proposed different prompt formats to enable self-debugging for diverse tasks and LLMs, and we demonstrated that the self-debugging ability can be taught with few-shot prompting, even if the model is not specially tuned for zero-shot debugging. For example, with code-davinci-002 that is not instruction tuned, we showed that self-debugging with few-shot prompting achieves similar or better performance gain than GPT-4.

Reviewer comment (Experiments):
The paper has shown that self-debugging is effective in fixing grammar and semantic error. Is it effective in fixing logic error? For example, the original implementation is correct but does not actually solve the problem. Will self-debugging be able to identify and fix such type of error?

Author response:
For all tasks, our evaluation metric is the execution accuracy, which considers the predicted code to be correct when it passes the given unit tests or its outputs match those of the ground truth code. Therefore, when we only use unit tests to determine whether to continue the self-debugging process, e.g., for Transcoder experiments where all unit tests are available, programs that pass the unit tests but have other logic errors would not be detected. On the other hand, when we request the LLM to determine the correctness of the generated code, e.g., on Spider and MBPP, the LLM can correct such logical errors to some extent. In our experiments, we manually investigated tens of correct predictions from each benchmark, and we did not notice such false positives.

Reviewer comment (Evaluation):
Improvement over baseline for Spider benchmark is only 2-3% which is not shown to be statistically significant. It could be accidental result of prompt change.

Author response:
We would like to note that 2-3% improvement on Spider is non-trivial. As shown in Figure 4 (a), without self-debugging, a 3% improvement from greedy decoding requires 16 samples. In particular, Figure 4 (b) shows that self-debugging improves the accuracy on the hardest SQL tasks by 9%, which is a significant boost.

Reviewer comment (Evaluation):
Same issue with code explanation without debugging for TransCoder and MBPP.

Author response:
First, we would like to clarify that code explanation results in Tables 1-3 are accompanied with self-debugging. The evaluation of code explanation without self-debugging is not our primary focus, but we presented the results in Figure 1 of Appendix E.1, where code explanation consistently improves the performance by 2-4%.

On the other hand, due to the availability of unit tests, we acknowledge that the improvement with explanation feedback on Trancoder and MBPP is generally less significant than on Spider. However, one benefit of explanation feedback is to provide more interpretable rationales on how LLMs infer the code correctness, and how the fixes are conducted. Such natural language feedback helps us to investigate the models’ weaknesses, and potentially can be used to finetune the model and improve its overall debugging and coding ability.

Furthermore, we would like to emphasize that one core contribution of our work is to conduct a comprehensive evaluation of self-debugging performance with different LLMs, applications, prompting methods and feedback formats, and demonstrate the importance of each factor. Specifically, both code explanations and traces are also commonly used for human debugging, which do not require any extra input from other human programmers. We show that stronger LLMs for code (code-davinci-002 with few-shot prompting and gpt-4) benefit more from such richer self-generated feedback, which can motivate future work on improving LLMs for investigating and understanding the semantic meaning of code.

Reviewer comment (Evaluation):
It seems that Self-Debugging without unit tests executions has very limited and possibly statistically insignificant improvements.

Author response:
As discussed in Section 5.2.2, we acknowledge that the improvement of self-debugging is less significant without unit test execution. However, even without unit tests, sometimes LLMs can still improve their performance with self-generated feedback. For example, as shown in Table 3b:
* With code-davinci-002, self-debugging without unit test execution improves the performance by up to 5%;
* GPT-4 without unit test execution improves the MBPP accuracy by 3.6%.

Furthermore, we would like to note that utilizing unit tests whenever applicable is also part of our self-debugging approach, and we consider the ability of leveraging execution of past generated programs as a key advantage of our self-debugging approach.

Reviewer comment (Presentation):
Section 4 is very hard to read. It constantly refers to Appendixes. Even with limited space, it is possible to present material much better so that paper would be self-contained and would not require readers to read appendixes to follow a whole section 4. In my opinion, if this is not fixed, this is a very serious issue with the paper.

Author response:
Thanks for the feedback about Section 4’s readability. We have added Figure 2 and 3 in the main paper to show the self-debugging prompt formats, which are abbreviated versions of the corresponding figures in the appendix. We hope this makes the section easier to read.

Reviewer comment (Presentation):
In table 1, what does line "Codex" represent? It is in the section "Self-Debugging (this work)", but the result is different from other lines in Self-Debugging section. Your paper does not explain what this line shows. Is this baseline without Self-Debugging techniques? If so, why is it presented in Self-Debugging section?

Author response:
Codex in Table 1 represents the baseline without self-debugging. We put it under the section “Self-Debugging (this work)” to denote that the following self-debugging results are built upon initial programs generated from this baseline. We added a divider around “Codex (baseline)” in Table 1 to make it clearer that these are the baseline results before self-debugging.

Reviewer comment (Presentation):
In table 2, Codex results for Spider correspond to Codex results for Spider in Table 1. However, Codex results for MBPP do not correspond to Codex results for MBPP in Table 1. Why?

Author response:
The Codex results in Table 2 were produced from greedy decoding, which demonstrate that self-debugging improves the performance with a higher sample efficiency. On the other hand, Codex baseline results in Table 1 were obtained with multiple samples, which demonstrate that self-debugging outperforms other baseline methods that leverage multiple samples. We updated the captions of Table 1 and 2 to clarify the settings.