Paper: LLM-Assisted Code Cleaning For Training Accurate Code Generators (ICLR-2024)

Reviewer comment (Experiments):
The proposed code transformations (i.e., renaming, modularizaing, and annotations) are a bit simple, and rely heavily on the capability of ChatGPT (GPT-3.5-TURBO). It is unknown whether the proposed method can be utilized for other, more powerful LLM-based code generation systems. For example, in Section 4.2.2, the authors acknowledge that the poor performance obtained on the planning dataset may stem from the model's inability to generate accurate annotations, indicating that the effectiveness of the proposed model depends on LLMs.

Author response:
We acknowledge the reviewer's point that the efficacy of our code transformation approach is influenced by the underlying language model employed. We selected GPT-3.5-Turbo for its balance between transformation accuracy and operational cost. To substantiate the versatility of our method, we have conducted an additional ablation study by substituting GPT-3.5-Turbo with the more recent GPT-4-Turbo model. Owing to the significantly higher cost of GPT-4-Turbo (approximately 8-10 times higher), we use a subset of the original dataset and refer to the modularized version as D_Modularize_. To ensure comparability, we analyzed the same subset of original and transformed programs for both the base and modularized datasets. Further details on the experimental setup can be found in Appendix Section C.2.


The table below illustrates that the D_Modularize_4 dataset yields improved performance over the D_Modularize dataset presented in the main paper. This enhancement not only demonstrates the adaptability but also the potential improvements of our approach using stronger models.

|                           | Pass@1   | Pass@10  | Pass@25 |
|---------------------------|----------|----------|---------|
| CL-7B+Base                | 16.3     | 31.6     | 37.6    |
| CL-7B+D_Modularize        | 18.8     | 33.0     | 38.2    |
| CL-7B+D_Modularize_4      | **19.4**   | **34.3**   | **40.0**  |

Reviewer comment (Evaluation):
The authors propose two steps of data cleaning (i.e., renaming, modularizaing) to the original source code. However, the authors did not validate the quality of the transformed code (the authors only tested whether the transformed code can be consistent to the original data). For example, whether the variable names actually became clearer and more readable after cleaning, and whether the model accurately segmented the code into modules. Hence, it is unknown whether such simple transformation process can enhance the quality of the training data.

Author response:
We appreciate the reviewer's emphasis on the importance of quantifying code quality. We have conducted a thorough analysis of the transformed dataset, as detailed in Section 4.1, where we offer both qualitative and quantitative evidence of quality enhancement. Below, we summarize the insights from Section 4.1 that pertain to code *quality*:

- Post-transformation, the median number of new functions introduced by the modularization process is three. This indicates that the modularization procedure actively decomposes existing programs to insert new, potentially more coherent functions.
- The helper functions created during modularization typically embody standard algorithms frequently encountered in competitive programming, such as `dfs`, `build_graph`, `gcd`, `dp`, and `binary_search`. This suggests that the new helper functions are meaningful and positively contribute to the overall code.
- A qualitative analysis of approximately 100 programs, including successful and unsuccessful transformations is presented with various success and failure modes detailed.

Moreover, we posit that the quality of the dataset can be indirectly inferred from the improved performance of models trained on it. The performance gains achieved through our modularization approach satisfy this measure of quality.

Based on reviewers’ concerns, we have additionally performed an analysis of the generated dataset using GPT-4 as a judge, a method gaining traction for assessing outputs from free-form language models [1,2]. While this approach might have subtle limitations, it still offers valuable insights to address the concerns raised. We tasked GPT-4 to quantitatively assess the transformed programs in regards to the meaningfulness of the transformations and consistency of original and transformed programs (i.e. following the same program logic and structure). Specifically, we asked GPT-4 to rate the transformation on 

- variable names (rating between 1-3 corresponding to not helpful, somewhat helpful, very helpful)
- function decomposition (rating between 1-3 corresponding to not helpful, somewhat helpful, very helpful)
- program consistency (rating between 0-1 corresponding to inconsistent and consistent)

Appendix C.1 provides prompts used for GPT-4 jude.  Analysis of about 1000 programs reveals that GPT-4 regards 99% of the transformations as helpful (of which only 3-5% of examples can judged as *can do better*). Similarly, over 99% of the transformed programs are judged as consistent with the original program. The following table provides a distribution of scores annotated

| | Score distribution | Average |
|---------------------------|----------|----------|
|Variable names | {3 : 967, 2 : 28, 1 : 3} | 2.96 |
| Function decomposition | {3 : 938, 2 : 59, 1 : 1} | 2.93 |
| Consistency | {1 : 994, 0 : 4} | 0.994 |

Appendix C.1 provides a qualitative analysis of GPT-4 scores and more details. 

Finally, we will release our generated dataset for further analysis for the community.

Reviewer comment (Novelty):
Code generation could be achieved by prompting a general LLM such as ChatGPT directly as well. Also, it has been found that by improving the prompts, the code generation performance can be improved. The authors may discuss this approach to accurate code generation: Liu et al., Improving ChatGPT Prompt for Code Generation, https://arxiv.org/abs/2305.08360

Author response:
We thank the reviewer for suggesting the related work. We have included more detailed related work in code generation with LLMs now. 

However, we would like to re-emphasize that in this work our goal is to study and propose methods for improving data quality for code generation. Data quality is an important aspect of training LLMs. We hope our proposed approach and findings will be useful for improving the training process for future (code) language models.

Reviewer comment (Evaluation):
How is the quality of the transformed code?

Author response:
We appreciate the reviewer's emphasis on the importance of quantifying code quality. We have conducted a thorough analysis of the transformed dataset, as detailed in Section 4.1, where we offer both qualitative and quantitative evidence of quality enhancement. Below, we summarize the insights from Section 4.1 that pertain to code *quality*:

- Post-transformation, the median number of new functions introduced by the modularization process is three. This indicates that the modularization procedure actively decomposes existing programs to insert new, potentially more coherent functions.
- The helper functions created during modularization typically embody standard algorithms frequently encountered in competitive programming, such as `dfs`, `build_graph`, `gcd`, `dp`, and `binary_search`. This suggests that the new helper functions are meaningful and positively contribute to the overall code.
- A qualitative analysis of approximately 100 programs, including successful and unsuccessful transformations is presented with various success and failure modes detailed.

Moreover, we posit that the quality of the dataset can be indirectly inferred from the improved performance of models trained on it. The performance gains achieved through our modularization approach satisfy this measure of quality.

Based on reviewers’ concerns, we have additionally performed an analysis of the generated dataset using GPT-4 as a judge, a method gaining traction for assessing outputs from free-form language models [1,2]. While this approach might have subtle limitations, it still offers valuable insights to address the concerns raised. We tasked GPT-4 to quantitatively assess the transformed programs in regards to the meaningfulness of the transformations and consistency of original and transformed programs (i.e. following the same program logic and structure). Specifically, we asked GPT-4 to rate the transformation on 

- variable names (rating between 1-3 corresponding to not helpful, somewhat helpful, very helpful)
- function decomposition (rating between 1-3 corresponding to not helpful, somewhat helpful, very helpful)
- program consistency (rating between 0-1 corresponding to inconsistent and consistent)

Appendix C.1 provides prompts used for GPT-4 jude.  Analysis of about 1000 programs reveals that GPT-4 regards 99% of the transformations as helpful (of which only 3-5% of examples can judged as *can do better*). Similarly, over 99% of the transformed programs are judged as consistent with the original program. The following table provides a distribution of scores annotated

| | Score distribution | Average |
|---------------------------|----------|----------|
|Variable names | {3 : 967, 2 : 28, 1 : 3} | 2.96 |
| Function decomposition | {3 : 938, 2 : 59, 1 : 1} | 2.93 |
| Consistency | {1 : 994, 0 : 4} | 0.994 |

Appendix C.1 provides a qualitative analysis of GPT-4 scores and more details. 

Finally, we will release our generated dataset for further analysis for the community.

Reviewer comment (Experiments):
Does the effectiveness of the proposed model depend on LLMs such as ChatGPT?

Author response:
We acknowledge the reviewer's point that the efficacy of our code transformation approach is influenced by the underlying language model employed. We selected GPT-3.5-Turbo for its balance between transformation accuracy and operational cost. To substantiate the versatility of our method, we have conducted an additional ablation study by substituting GPT-3.5-Turbo with the more recent GPT-4-Turbo model. Owing to the significantly higher cost of GPT-4-Turbo (approximately 8-10 times higher), we use a subset of the original dataset and refer to the modularized version as D_Modularize_. To ensure comparability, we analyzed the same subset of original and transformed programs for both the base and modularized datasets. Further details on the experimental setup can be found in Appendix Section C.2.


The table below illustrates that the D_Modularize_4 dataset yields improved performance over the D_Modularize dataset presented in the main paper. This enhancement not only demonstrates the adaptability but also the potential improvements of our approach using stronger models.

|                           | Pass@1   | Pass@10  | Pass@25 |
|---------------------------|----------|----------|---------|
| CL-7B+Base                | 16.3     | 31.6     | 37.6    |
| CL-7B+D_Modularize        | 18.8     | 33.0     | 38.2    |
| CL-7B+D_Modularize_4      | **19.4**   | **34.3**   | **40.0**  |

Reviewer comment (Experiments):
The experiment section of the paper reports numbers on the subsets of two datasets. It would be nice to clearly outline the filtering criteria that are used for each dataset. I have certain questions regarding this in the “Clarifications section”.

Author response:
No additional filtering was performed for APPS. Further details are added to the appendix

Reviewer comment (Evaluation):
In many of the tables, there has been no reference or explanation to numbers that show a negative effect on results. For example, on the Code-Contest dataset, D_rename works worse than the baseline. It would be nice if the authors could be candid about this in their writing.

Author response:
We thank the reviewer for the feedback and have updated the manuscript to reflect this.

Reviewer comment (Presentation):
I believe that the CL-7B + D_distill number is missing in table 4(a). Can you please include that number in the rebuttal?

Author response:
Fixed and updated in manuscript.

Reviewer comment (Evaluation):
The improvements on Code-Contests seem to be not as effective as would be expected.

Author response:
While CodeContests improvements do not seem significant in magnitude, we want to emphasize that the CodeContests dataset is an even more challenging dataset in comparison to APPS and thus also harder to improve upon. For context, within the AlphaCode models, the 41B model does not improve notably over the 9B model, suggesting that even scaling has a limited effect. To our knowledge, the only other study that has tackled this dataset without closed-source API-gated models is “outline-generate” [7]. They proposed a tokenization approach and observed similar improvements against AlphaCode.

Reviewer comment (Presentation):
It’s not immediately obvious how the natural language “plans” are used. My initial understanding was that they are provided as a docstring for each function. It would be nice to clarify in Fig 1 that they are prepended to the program as a comment.

Author response:
Fixed and updated in manuscript.

Reviewer comment (Experiments):
In the APPs benchmark, do you consider all problems from “codeforces”, “codechef” and “atcoder”? Or is there some further filtering done after that? If further filtering has been done, can you please clarify what procedure has been followed?

Author response:
No additional filtering was performed for APPS. Further details are added to the appendix

Reviewer comment (Writing):
What does the line “These cases are sometimes due to incorrect programs but more …. while only a single test solution is provided” mean?

Author response:
This refers to the observation that certain problems admit multiple correct solutions that are not equivalent to one another. For instance, a problem may allow the output of a list of numbers in any order. Our examination of the datasets revealed that both APPS and CodeContests typically include test cases that cover only one of the many correct solutions (e.g., only one specific ordering of the list). Example - https://codeforces.com/problemset/problem/1512/B.

Reviewer comment (Evaluation):
The planning results are also modest, but this is interesting in its own right, and the analysis of how ground truth plans would help considerably is a good way to isolate much of the problem to the plan creation rather than plan execution.

Author response:
We agree with the reviewer that planning results are modest/mixed bag. To add additional context and perspective – improving the reasoning and planning capabilities of language models is a challenging problem and a research direction actively explored by the community [10]. Our experiment of disentangling plan generation with code implementation shows that even after training on such plans, current models cannot generalize and construct plans for new challenging problems. To provide the insights more clearly, we have added some additional context in the paper summarized below: 

We adopted the supervised learning paradigm, aiming to refine planning by training on natural language plans curated using our methodology akin to prior works [4,5,6]. While the results from D_planning are mixed, we believe they raise critical questions for future research. Two particularly promising avenues are:

- Defining "quality" for natural language plans: The mixed success with planning might be attributed to inaccuracies in the automatically generated plans. Future data curation techniques that filter or augment this precision (akin to entailment models used in [2]) is promising.

- Alternative learning paradigms for enhancing planning: The supervised learning paradigm followed in this work might be insufficient for models to generalize planning in complex domains. Future work can explore alternative learning algorithms, possibly over our modularization approach which naturally decomposes programs usually beneficial for solving complex problems.

Reviewer comment (Experiments):
While most of the paper was easy to read, I was quite unclear on the distillation dataset baseline – see the Questions section for details.

Author response:
Yes, in the distillation experiment, we take the problem statements from the training dataset and use the GPT-3.5-Turbo model to generate multiple, *diverse* solutions using a few-shot prompting approach. Next, we filter the generated solutions for correctness based on the ground truth test cases provided in the dataset to ensure we are not training on incorrect hallucinated solutions. This experiment is meant to provide a strong baseline that distills larger-model-generated outputs which has proven to be a very popular approach for both instruction tuning and task-specific finetuning [6,8,9]. 
In contrast to distillation, our transformation approach enjoys the benefits of performing the simpler operation of transforming (editing) existing solutions instead of directly generating them. Thus it yields about 94\% successful transformation rate whilst the distillation baseline can only generate solutions for about 50\% of the problems.

Reviewer comment (Presentation):
In Table 3 there is one missing entry – Pass@1 APPS Interview Distill. Where is it?

Author response:
We have fixed these issues and updated the paper.

Reviewer comment (Experiments):
I don't understand the "distill" baseline dataset laid out at the end of 3.2, and referenced at various points.

Author response:
Yes, in the distillation experiment, we take the problem statements from the training dataset and use the GPT-3.5-Turbo model to generate multiple, *diverse* solutions using a few-shot prompting approach. Next, we filter the generated solutions for correctness based on the ground truth test cases provided in the dataset to ensure we are not training on incorrect hallucinated solutions. This experiment is meant to provide a strong baseline that distills larger-model-generated outputs which has proven to be a very popular approach for both instruction tuning and task-specific finetuning [6,8,9]. 
In contrast to distillation, our transformation approach enjoys the benefits of performing the simpler operation of transforming (editing) existing solutions instead of directly generating them. Thus it yields about 94\% successful transformation rate whilst the distillation baseline can only generate solutions for about 50\% of the problems.

Reviewer comment (Experiments):
My best guess is that you're doing synthetic data generation to generate a new dataset by prompting with few-shot examples from the modular dataset. Is it generating both the test cases *and* the solutions to them? Or are the test cases taken from somewhere and then it's just generating solutions?

Author response:
Yes, in the distillation experiment, we take the problem statements from the training dataset and use the GPT-3.5-Turbo model to generate multiple, *diverse* solutions using a few-shot prompting approach. Next, we filter the generated solutions for correctness based on the ground truth test cases provided in the dataset to ensure we are not training on incorrect hallucinated solutions. This experiment is meant to provide a strong baseline that distills larger-model-generated outputs which has proven to be a very popular approach for both instruction tuning and task-specific finetuning [6,8,9]. 
In contrast to distillation, our transformation approach enjoys the benefits of performing the simpler operation of transforming (editing) existing solutions instead of directly generating them. Thus it yields about 94\% successful transformation rate whilst the distillation baseline can only generate solutions for about 50\% of the problems.

Reviewer comment (Novelty):
Two relevant pieces of work on synthetic data generation of code for LLMs are the Self Taught Reasoner (STaR) (Zelikman et al 2022) and Language Models Can Teach Themselves to Program Better (Haluptzok et al 2022). Those would be relevant to reference under the "Synthetic data for LLMS" section of Related Work, and could also relate to the distillation (though as mentioned before, I understand the distill baseline less).

Author response:
Yes, these works are relevant. STaR learns natural language reasoning (corresponding to our planning approach) in simpler domains. Haluptzok et al generates datasets using language models and further models on them. We have added them to the related works.

Reviewer comment (Writing):
Typo at end of 2.1 with random sentence ending: "steps quite effectively. effective in generating high-quality outputs."

Author response:
We have fixed these issues and updated the paper.

Reviewer comment (Writing):
The sentence "We obtain three *parallel* datasets at the end of our cleaning process, one for each of renaming, modularization, and planning" and in particular the word "parallel" is a bit misleading since at least to me it suggests that each dataset comes from applying a single transformation to the original dataset independent of the others, but actually the 3 transformations build on each other. This is clarified by Table 2 but would be helpful to have in the text as well.

Author response:
We have fixed these issues and updated the paper.

Reviewer comment (Presentation):
Table 4 isn't actually labelled "Table 4" anywhere (since there's no shared caption).

Author response:
We have fixed these issues and updated the paper.