Paper: Is Self-Repair a Silver Bullet for Code Generation? (ICLR-2024)

Reviewer comment (Experiments):
The scope of the study is limited to solving isolated programming challenges while ignoring real-world development. The study focuses completely on programming challenge datasets, such as HumanEval and APPs, which have several characteristics that are not realistic in daily development. Therefore, while the initial conclusions of the paper are appreciated, these takeaways might not be applicable to more realistic scenarios.

Author response:
*The scope of our study is limited to self-contained Python programs, which is quite different from real-world software engineering.*

Software Engineering (SE) and competitive programming both present their own unique challenges. SE tasks often involve incomplete  task specifications as well as (long) contextual dependencies. Competitive programming, on the other hand, is mainly challenging due to the logical and algorithmic complexity, often requiring the use of dynamic programming and graph algorithms to solve the task.

An ideal self-repair model should be able to (1) fix errors in logically complex tasks, (2) handle ambiguous specifications and missing context, and (3) repair without a clear test oracle. The community would benefit from in-depth studies of each capability. In this paper, we isolate the first aspect, providing insights that would be hard to untangle otherwise.

With this in mind, we believe competitive programming becomes a well suited testbed for our analysis:
- Using logically complex, well-specified programming puzzles to benchmark models has a rich history in the literature [0,1,2].
- Recent work shows that even contemporary models are still challenged by intermediate/advanced-level competitive programming tasks [3, 4].
- Competitive programming tasks have relatively complete specifications and unit tests, so we do not have to worry as much about our results being muddled by degenerate cases (e.g. the solution is incorrect simply because the task specification did not provide a sufficient definition of an API).

In our study, we are thus able to hone in on self-repair's efficacy in algorithmically challenging tasks of varying difficulty levels. We use this primarily to isolate the importance of the feedback stage, but it also enables us to discover non-obvious relationships such as the non-straightforward interplay between task difficulty and self-repair efficacy (Section 4.1 and Appendix B). This surprising fact would have been difficult to discover without isolating the effect of the logical/algorithmic complexity of the task at hand.

In future work, we'd like to explore the other two capabilities as well. These are motivated by the question of how future software engineering workflows should best leverage AI code generation tools. One can imagine workflows which emphasize encapsulation to an extent that each individual piece is no more complex than a programming puzzle, but tricky bugs still tend to arise at the interface level. Furthermore, the fact that Test-Driven Development has fallen out of vogue suggests that developers do not like writing unit tests first, which motivates capability (3). Similarly, although natural language may not have played a big role in software engineering historically, minimizing the impact of ambiguity in NL specifications is now necessary in order to enable capability (2).

In summary, as we hope this discussion has shown, we wholeheartedly agree with the reviewer that there is much interesting work left to be done in this space. Furthermore, **we will revise the Introduction and Future Work sections to clarify the scope of our contributions, as well as highlight some of the nuances discussed above**.

Reviewer comment (Experiments):
The samples in the studied datasets are provided with clear and complete problem descriptions, which are not always available in real-world programming practice. In programming challenges, the expected functionality of the program is fully revealed in the prompt, allowing the model to maximize diversity within a constrained semantic space for pass@k generation. However, in ongoing development, developers often start with high-level goals, and human intent is not explicitly specified before prompting the code LM. In such cases, execution feedback from unit tests provides critical guidance for iterative self-repair, which is absent for top-k generation. Therefore, the conclusion that pass@k is comparable to or better than self-repair in programming challenges may not hold in daily development and requires further study.

Author response:
*The scope of our study is limited to self-contained Python programs, which is quite different from real-world software engineering.*

Software Engineering (SE) and competitive programming both present their own unique challenges. SE tasks often involve incomplete  task specifications as well as (long) contextual dependencies. Competitive programming, on the other hand, is mainly challenging due to the logical and algorithmic complexity, often requiring the use of dynamic programming and graph algorithms to solve the task.

An ideal self-repair model should be able to (1) fix errors in logically complex tasks, (2) handle ambiguous specifications and missing context, and (3) repair without a clear test oracle. The community would benefit from in-depth studies of each capability. In this paper, we isolate the first aspect, providing insights that would be hard to untangle otherwise.

With this in mind, we believe competitive programming becomes a well suited testbed for our analysis:
- Using logically complex, well-specified programming puzzles to benchmark models has a rich history in the literature [0,1,2].
- Recent work shows that even contemporary models are still challenged by intermediate/advanced-level competitive programming tasks [3, 4].
- Competitive programming tasks have relatively complete specifications and unit tests, so we do not have to worry as much about our results being muddled by degenerate cases (e.g. the solution is incorrect simply because the task specification did not provide a sufficient definition of an API).

In our study, we are thus able to hone in on self-repair's efficacy in algorithmically challenging tasks of varying difficulty levels. We use this primarily to isolate the importance of the feedback stage, but it also enables us to discover non-obvious relationships such as the non-straightforward interplay between task difficulty and self-repair efficacy (Section 4.1 and Appendix B). This surprising fact would have been difficult to discover without isolating the effect of the logical/algorithmic complexity of the task at hand.

In future work, we'd like to explore the other two capabilities as well. These are motivated by the question of how future software engineering workflows should best leverage AI code generation tools. One can imagine workflows which emphasize encapsulation to an extent that each individual piece is no more complex than a programming puzzle, but tricky bugs still tend to arise at the interface level. Furthermore, the fact that Test-Driven Development has fallen out of vogue suggests that developers do not like writing unit tests first, which motivates capability (3). Similarly, although natural language may not have played a big role in software engineering historically, minimizing the impact of ambiguity in NL specifications is now necessary in order to enable capability (2).

In summary, as we hope this discussion has shown, we wholeheartedly agree with the reviewer that there is much interesting work left to be done in this space. Furthermore, **we will revise the Introduction and Future Work sections to clarify the scope of our contributions, as well as highlight some of the nuances discussed above**.

Reviewer comment (Experiments):
The samples in the studied datasets are mostly short and self-contained, lacking the complicated data and inter-procedural dependencies found in real-world software. Pass@k explores token diversity without directional guidance, but the search space increases exponentially with program length. Real-world software often contains hundreds or thousands of lines of code and complex dependencies, making blind generation of k sequences less effective. Execution feedback can help quickly locate problems and focus fixes. Therefore, for more complex programs, further study is needed to determine whether self-repair is equivalent to top-k generation.

Author response:
*The scope of our study is limited to self-contained Python programs, which is quite different from real-world software engineering.*

Software Engineering (SE) and competitive programming both present their own unique challenges. SE tasks often involve incomplete  task specifications as well as (long) contextual dependencies. Competitive programming, on the other hand, is mainly challenging due to the logical and algorithmic complexity, often requiring the use of dynamic programming and graph algorithms to solve the task.

An ideal self-repair model should be able to (1) fix errors in logically complex tasks, (2) handle ambiguous specifications and missing context, and (3) repair without a clear test oracle. The community would benefit from in-depth studies of each capability. In this paper, we isolate the first aspect, providing insights that would be hard to untangle otherwise.

With this in mind, we believe competitive programming becomes a well suited testbed for our analysis:
- Using logically complex, well-specified programming puzzles to benchmark models has a rich history in the literature [0,1,2].
- Recent work shows that even contemporary models are still challenged by intermediate/advanced-level competitive programming tasks [3, 4].
- Competitive programming tasks have relatively complete specifications and unit tests, so we do not have to worry as much about our results being muddled by degenerate cases (e.g. the solution is incorrect simply because the task specification did not provide a sufficient definition of an API).

In our study, we are thus able to hone in on self-repair's efficacy in algorithmically challenging tasks of varying difficulty levels. We use this primarily to isolate the importance of the feedback stage, but it also enables us to discover non-obvious relationships such as the non-straightforward interplay between task difficulty and self-repair efficacy (Section 4.1 and Appendix B). This surprising fact would have been difficult to discover without isolating the effect of the logical/algorithmic complexity of the task at hand.

In future work, we'd like to explore the other two capabilities as well. These are motivated by the question of how future software engineering workflows should best leverage AI code generation tools. One can imagine workflows which emphasize encapsulation to an extent that each individual piece is no more complex than a programming puzzle, but tricky bugs still tend to arise at the interface level. Furthermore, the fact that Test-Driven Development has fallen out of vogue suggests that developers do not like writing unit tests first, which motivates capability (3). Similarly, although natural language may not have played a big role in software engineering historically, minimizing the impact of ambiguity in NL specifications is now necessary in order to enable capability (2).

In summary, as we hope this discussion has shown, we wholeheartedly agree with the reviewer that there is much interesting work left to be done in this space. Furthermore, **we will revise the Introduction and Future Work sections to clarify the scope of our contributions, as well as highlight some of the nuances discussed above**.

Reviewer comment (Experiments):
It is not clear whether fine-tuning for self-repair could easily overcome the weaknesses identified. The study examines LLM self-repair capability only through prompting, without model parameter optimization for self-repair. It remains uncertain whether inexpensive fine-tuning could enable the model to more effectively understand and use feedback. Concluding that self-repair is not a silver bullet without attempting straightforward fine-tuning may be too strong.

Author response:
*Our study is limited to prompting strategies, and does not consider fine-tuning.*

We certainly agree that such an analysis would be of interest to the community! However, we believe that this falls outside the scope of this study, which we emphasize is already significant in terms of its depth (and cost). In addition to the methodological challenges, fine-tuning also comes with practical concerns such as cost and model availability. We thus leave it to future work to investigate whether fine-tuning can alleviate the feedback bottleneck identified by this paper. Besides, prompting may be accessible to a wider audience than fine-tuned models in the current AI ecosystem; studying self-repair in this context may therefore benefit AI practitioners more in practice.

Reviewer comment (Evaluation):
The findings around overall efficacy compared to baseline somewhat contradict with the results from Chen et. al. 2023b, which show that self-repair could provide significant increase in sample efficiency.

Author response:
We agree with the reviewer that the relationship to Chen et al. (2023b) is important to discuss. After replying to the reviewer’s specific questions below, we will give a detailed account of how our work compares to that of Chen et al (2023b).

*Detailed comparison to Chen et al. (2023b).*

As mentioned in the Related Work, Chen et al. (2023b)’s method is indeed closely related to ours; both use few-shot prompting to encourage the model to first retrospect on the code (in order to understand why it failed) and then perform repair. There are, however, a few differences.

1. The main reason our results differ from those of Chen et al. (2023b) is that we conduct our experiments in a slightly different setting.

In Chen et al.’s study, the self-debugging method has access to a correctness oracle to decide whether to proceed with debugging or not; the baseline does not have access to the oracle. In our work, both the baseline and the self-repair approach have equal access to an oracle (when evaluating pass@k) and are compared with the same sample budget. Thus, the performance improvement over the baseline is more prominent in Chen et al. (2023b)’s work than in ours.

We choose to grant both baseline and repair models equal access to the oracle so that we can better analyze the tradeoff between increased sampling overhead and accuracy gained. Chen et al. (2023b) instead aim to show how oracles and different repair strategies can improve the final accuracy compared to current, standard, non-repair-based approaches. Thus, our findings are not actually contradictory to those of Chen et al. (2023b).

2. Generally speaking, our work focuses on exploring the efficacy of self-repair in depth, while Chen et al. focus on comparing a broader range of different self-debugging strategies in a few different domains. Our studies thus complement each other.

Concretely, our work focuses on what Chen et al. calls the “Unit Test + Explanation” style of feedback, in which the model is provided with external feedback from a unit test suite but then has to generate its own explanation of why the test failed. We study the effect of this textual explanation in great detail through our experiments in sections 4.2 and 4.3, and show that it is the limiting factor in this type of self-repair. We also offer a substantive discussion of the effect of the hyper-parameters, which amongst other things highlights the need to obtain sufficiently diverse initial samples if self-repair is to be successful.

The scope of our study is also limited to what Chen et al. call “Text-to-Python Generation” tasks, while they also consider code translation and SQL query generation tasks. However, we do so in both an easier setting (HumanEval) and a significantly more challenging setting where baseline performance is lower (APPS), while Chen et al. focus on the relatively easy MBPP dataset. As we show in section 4.1 and Appendix B, the relationship between task difficulty and self-repair performance is quite subtle, an insight which we would not have been able to show without analyzing multiple datasets in detail.

Reviewer comment (Experiments):
Although the ‘self-debugging’ work from Chen et. al. is mentioned in the related work, more comparative analysis would strengthen the claim of this paper.

Author response:
We agree with the reviewer that the relationship to Chen et al. (2023b) is important to discuss. After replying to the reviewer’s specific questions below, we will give a detailed account of how our work compares to that of Chen et al (2023b).

*Detailed comparison to Chen et al. (2023b).*

As mentioned in the Related Work, Chen et al. (2023b)’s method is indeed closely related to ours; both use few-shot prompting to encourage the model to first retrospect on the code (in order to understand why it failed) and then perform repair. There are, however, a few differences.

1. The main reason our results differ from those of Chen et al. (2023b) is that we conduct our experiments in a slightly different setting.

In Chen et al.’s study, the self-debugging method has access to a correctness oracle to decide whether to proceed with debugging or not; the baseline does not have access to the oracle. In our work, both the baseline and the self-repair approach have equal access to an oracle (when evaluating pass@k) and are compared with the same sample budget. Thus, the performance improvement over the baseline is more prominent in Chen et al. (2023b)’s work than in ours.

We choose to grant both baseline and repair models equal access to the oracle so that we can better analyze the tradeoff between increased sampling overhead and accuracy gained. Chen et al. (2023b) instead aim to show how oracles and different repair strategies can improve the final accuracy compared to current, standard, non-repair-based approaches. Thus, our findings are not actually contradictory to those of Chen et al. (2023b).

2. Generally speaking, our work focuses on exploring the efficacy of self-repair in depth, while Chen et al. focus on comparing a broader range of different self-debugging strategies in a few different domains. Our studies thus complement each other.

Concretely, our work focuses on what Chen et al. calls the “Unit Test + Explanation” style of feedback, in which the model is provided with external feedback from a unit test suite but then has to generate its own explanation of why the test failed. We study the effect of this textual explanation in great detail through our experiments in sections 4.2 and 4.3, and show that it is the limiting factor in this type of self-repair. We also offer a substantive discussion of the effect of the hyper-parameters, which amongst other things highlights the need to obtain sufficiently diverse initial samples if self-repair is to be successful.

The scope of our study is also limited to what Chen et al. call “Text-to-Python Generation” tasks, while they also consider code translation and SQL query generation tasks. However, we do so in both an easier setting (HumanEval) and a significantly more challenging setting where baseline performance is lower (APPS), while Chen et al. focus on the relatively easy MBPP dataset. As we show in section 4.1 and Appendix B, the relationship between task difficulty and self-repair performance is quite subtle, an insight which we would not have been able to show without analyzing multiple datasets in detail.

Reviewer comment (Experiments):
Is there a specific reason to restrict feedback and repair samples to 1 in the analysis of feedback boosting (section 4.2)?

Author response:
> Is there a specific reason to restrict feedback and repair samples to 1 in the analysis of feedback boosting (section 4.2)?

This restriction is mainly a practical one: separating the feedback and repair makes the experiment significantly more time-consuming (and costly) to run, since each stage must now be implemented as a separate API call. With this restriction, we can control the cost of this experiment by using a smaller value of $N_f$ and $N_r$ without increasing the risk of statistical artifacts too much. We emphasize that the preceding experiments already showed that this setting is the most effective, and we therefore do not believe this limits the analysis in practice.

Reviewer comment (Experiments):
Should we expect similar results using the ‘self-debugging’ approach that uses few-shot prompts?

Author response:
> Should we expect similar results using the ‘self-debugging’ approach that uses few-shot prompts?

Although our results are already similar (and we do use few-shot prompting; see Appendix F for a complete list of the prompts we use), we agree that our results are–generally speaking–not as strongly in favor of self-repair/self-debugging. This is because of slight differences in the experimental setting being investigated; see the discussion below for a detailed comparison.