Paper: WizardCoder: Empowering Code Large Language Models with Evol-Instruct (ICLR-2024)

Reviewer comment (Evaluation):
In section 4.3, Table 1, Result of pass@1(%) on HumanEval and MBPP, the result of pass@1(%) on MBPP is not as good as the result on HumanEval. HumanEval only tests a narrow distribution and can be overfitted. The benchmark needs to be upgraded.

Author response:
We acknowledge the need for a more comprehensive set of benchmarks in the coding domain. Our evaluation extends beyond HumanEval, encompassing a diverse set of benchmarks, including MBPP, DS-1000, and MultiPl-E. Notably, when compared to SOTA open-source software (OSS) models, our models showcase substantial improvements across various benchmarks. Specifically, our models achieve approximately a 20-point improvement on HumanEval, a 5.2-point improvement on MBPP, a 7.4-point improvement on DS-1000 (insertion), and an average 9.3-point improvement on MultiPL-E. These significant enhancements underscore the effectiveness of our work. Additionally, it's worth noting that Codellama-instruct only achieves a 2-point improvement compared to Codellama on MBPP.

Reviewer comment (Reproducibility):
In section 3.2, TRAINING WizardCoder," We iteratively employ the Code Evol-Instruct technique on this dataset consisting of around 20k samples to produce evolved data." What is the number of iterations and the criteria for completion?

Author response:
We conducted a total of 4 iterations. Figure 3a reveals that the optimal performance is achieved after 3 rounds of evolutions on the dev set. Based on this observation, we made the decision to discontinue further iterations.

Reviewer comment (Experiments):
In section 3, when iterating the Code Eval-Instruct on the sample dataset, what model is used to generate evolved data? Does the difference in the quality of pre-trained model outputs lead to different results?

Author response:
Please refer to the section "2. Common Response to using different evolution execution models" in the Overall Response. We have provided the results of using GPT-4 as the evolution execution model.

We have mentioned in Section 4.2 that we use GPT3.5 (gpt3.5-turbo) to generate evolved data. We conduct additional experiments, replacing GPT3.5 (gpt3.5-turbo) with GPT-4 to generate evolved rounds. HumanEval Pass@1 scores increased from 73.2 to 73.8 for 34B and from 59.8 to 62.2 for 15B. Using a superior LLM does contribute to better evolved data; however, the improvement is not strongly correlated with the coding performance of the evolution executing model (88.4 for GPT4 and 73.2 for GPT3.5).

Reviewer comment (Theory):
It is not clear how the authors came up with the list of heuristics for data evolution. This unclarity makes such approaches less applicable to wide range of tasks.

Author response:
We created these five heuristic methods to increase the complexity of code instructions, aligning with the characteristics of the code domain and previous research on code models' limitations. The first three heuristics naturally escalate coding task difficulty by incorporating additional requirements, replacing common requirements, or introducing extra reasoning steps. This aligns with observed trends on platforms like LeetCode, where tasks with more requirements or intricate reasoning are considered more challenging. The fourth heuristic introduces erroneous code as an adversarial sample, inspired by prior research in attacking pre-trained code models [1][2], serving to assess the robustness of the code LLM. The fifth heuristic, emphasizing time and space complexity, aligns with the significance of optimization in the coding domain. Previous studies [3] indicate that tasks requiring optimized program performance pose heightened difficulty, making this direction a valuable avenue for complexity improvement.

[1] Yang, Zhou et al. “Natural Attack for Pre-trained Models of Code.” 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE).

[2] Jha, Akshita et al. “CodeAttack: Code-based Adversarial Attacks for Pre-Trained Programming Language Models.” AAAI 2022.

[3] Madaan, A. et al. "Learning Performance-Improving Code Edits." (arxiv, 2023)

Reviewer comment (Experiments):
While the ablation studies in the main body provides some insights on the efficacy of the technique (additional clarification in the questions/recommendation section).

Author response:
In the upper part of Table 4, the Pass@1 scores exhibit fluctuations rather than a monotonic decrease; it's important to highlight that only the fourth round notably deteriorates. This decline may be attributed to two key factors. First, continual evolution could lead to overly complicated instructions, rendering them unreasonable. Second, beyond a certain round, instructions may become excessively complex, posing a significant challenge to the evolution executing model (i.e., gpt3.5-turbo) in generating suitable responses.

In the lower part of Table 4, **the decreasing scores are normal**. As instructions become more complex, the length of samples increases, leading to a reduction in the number of samples in each round to control for the same token count.

It's crucial to recognize that individual metrics in Table 4 for a specific round with complex data might not fully represent the overall impact on the final model performance. Figure 3 provides insights, indicating that merging different complex-level data can result in performance gains.

Reviewer comment (Novelty):
While the idea is interesting, but it seems very incremental compared to prior work and the contributions are limited.

Author response:
Please refer to "1. Common Response to novelty and contributions" in the Overall Response regarding the novelty and contributions of our work.

Reviewer comment (Experiments):
Update: based on the author's feedback, the coding solutions in the training data are from GPT-3.5-turbo. This presents a risk of data leakage, that WizardCoder becomes an implicitly distilled model of GPT-3.5-turbo on a specific capability.

Author response:
Thank you for taking the time to review our rebuttals, and we appreciate your ongoing consideration of our submission.

We have conducted additional experiments to replace the evolution execution models with the same family of our models. The results are presented in the "Overall Response: 2. Common Response to using different evolution execution models". For your convenience, we also include some results here:

|  Base Model  | Evol Model | HumanEval    |
| ----- |----| ---- |
| CodeLlama-34B | GPT3.5(ChatGPT) | 73.2 |
| CodeLlama-34B | CodeLlama | 70.1 |

As you suggested, we explored using the same family of open-sourced models (OSS), specifically the CodeLlama-Instruct-34B, for executing the entire process of Code Evol-Instruct. The OSS-base Code Evol-Instruct tuned models also exhibited performance improvements compared to the code-alpaca fine-tuned model, increasing from 65.9 to 70.1. **This demonstrates that our proposed method is still effective without GPT-3.5.**
 
To eliminate any possible misunderstandings, we are willing to explain our experimental setup for you in detail. Our experimental setup is reasonable lies in that our baseline code-alpaca is also generated from the GPT3.5 model but using a different method (i.e., self-instruct). As shown in the upper part of Table 4 of Section 5 in the main submission, when ensuring a similar amount of training data, the model (rounds 1-4) fine-tuned from Code Evol-Instruct created data is superior to the model (rounds 0) fine-tuned with code-alpaca data.
 
Overall, our Code Evol-Instruct emerges as the pivotal factor in driving performance enhancement, rather than some implicit data leakage from GPT-3.5. We sincerely hope that the clarification could address your concern.

Reviewer comment (Evaluation):
Weak analysis. Why does round 4 have a negative impact on the quality? The paper mentioned "some evolution process will fail" but provided no detail. Other analysis related questions will be in the "questions" section.

Author response:
Our empirical results reveal a potential saturation point from Code Evol-Instruct after a specific number of rounds, namely 4. This decline may be attributed to two key factors. First, continual evolution could lead to overly complicated instructions, rendering them unreasonable. Second, beyond a certain round, instructions may become excessively complex, posing a huge challenge to the evolution executing model (i.e., gpt3.5-turbo) to generate suitable responses.

Reviewer comment (Writing):
Writing quality can be improved. The paper uses more sentences to repeatedly claim WizardCoder's performance (which is indeed impressive) but vague on implementation details and analysis.

Author response:
We present the implementation details in Section 4.2, with the evolution prompts discussed in Section 3.1 and the training process outlined in Section 3.2. The prompt formats are provided in Appendix A, baseline details in Appendix B, and data filtering details in Appendix C.

Furthermore, our analysis in Section 5 encompasses various aspects, such as the impact of different numbers of evolution rounds, the relationships between complexity and quantity, and the correlations between complexity and similarity.

Reviewer comment (Reproducibility):
Where does the new coding solution come after expanding the instruction?  Did we use the base model itself, or GPT-3 to generate them?

Author response:
We use the same model (gpt3.5-turbo) to generate the code solutions.

Reviewer comment (Reproducibility):
Did we do any deduplication of the expanded instructions?

Author response:
The seed data, employing the self-instruct technique, is inherently deduplicated. Given that the expanded instructions derive from distinct coding tasks, there is no need for deduplication in the subsequent evolved process.

Reviewer comment (Experiments):
Did we verify the quality of new instructions by executing the code generated?

Author response:
We have tried to execute all the generated Python codes. All of them can run without any compile errors.

Reviewer comment (Presentation):
Figure 1 lower plot. It's better to use pattern instead of color to separate two data classes to make it more friendly for color-blindness.

Author response:
Thank you for your suggestion. We updated Figure 1 lower plot with different patterns.

Reviewer comment (Writing):
Section 1 paragraph 2. Reviewer thinks "Code" need not to have capitalized first letter.

Author response:
Thank you for your suggestion. We updated Section 1 paragraph 2.

Reviewer comment (Novelty):
Section 2 Related Work. We should also mention CodeLlama's self-instruct work and compare the differences. https://arxiv.org/abs/2308.12950

Author response:
We have mentioned the CodeLlama-Series models in the related work. The CodeLlama-Instruct model is also one of the baseline models in our experiments. We include extra discussion of this model in the related works section.

Reviewer comment (Writing):
Page 4, Code Evolution Heuristic Methods table. Please explain the 4th instruct's purpose and examples of expansion. "Provide a piece of erroneous code as a reference to increase misdirection". It's not clear to the Reviewer what this is doing and why it would be effective.

Author response:
For a specific instruction, this method includes an error code in it. For example, the original instruction is: `Write a Python function to tell me what the date is today.`
   
The evolved instruction is: 
`Write a Python function to tell me what the date is today, using the datetime module. The function should return a string in the format of “Today is YYYY-MM-DD”. For example, if today is November 17, 2023, the function should return “Today is 2023-11-17”. As a reference, here is a piece of erroneous code that does not work as intended:`
```python
import datetime
def get_date():
    today = datetime.date.today()
    return "Today is " + today
```

This erroneous code serves as an adversarial sample designed to challenge the code LLM, akin to previous works focused on attacking pre-trained code models. [1][2]

[1] Yang, Z., Shi, J., He, J., & Lo, D. (2022). Natural Attack for Pre-trained Models of Code. 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), 1482-1493.

[2] Jha, A., & Reddy, C.K. (2022). CodeAttack: Code-based Adversarial Attacks for Pre-Trained Programming Language Models. AAAI Conference on Artificial Intelligence.

Reviewer comment (Evaluation):
Page 4, Code Evolution Heuristic Methods table. Last prompt about increasing time/space complexity: how can we verify the response actually achieved it?

Author response:
This heuristic, which underscores time and space complexity, aligns with the importance of optimization in the coding domain. Previous studies [3] suggest that tasks demanding optimized program performance present heightened difficulty, rendering this direction a valuable avenue for complexity improvement. Ideally, all responses should achieve 100% compliance with the specified complexity requirements. Given our use of GPT3.5(gpt3.5-turbo) to generate responses, we can express a reasonable level of confidence that the responses are well-aligned with the stipulated requirements.

[3] Madaan, A., Shypula, A., Alon, U., Hashemi, M., Ranganathan, P., Yang, Y., Neubig, G., & Yazdanbakhsh, A. (2023). Learning Performance-Improving Code Edits. ArXiv, abs/2302.07867.

Reviewer comment (Writing):
Section 5 paragraph 1, "amalgamate the training set", what about using a simpler word like  "merge the training set"?

Author response:
Thank you. We updated it in Section 5 paragraph 1.

Reviewer comment (Writing):
Section 5 paragraph 2. The Reviewer assumes more rounds equals more complexity. But it's better to make it explicit and provide some examples to show what does complexity actually mean in this context.

Author response:
Yes. More rounds equals more complexity. We have provided some evol examples in Appendix D.

Reviewer comment (Presentation):
Section 5 "Complexity and Similarity". This paragraph didn't really talk about complexity, unless more rounds = more complexity?

Author response:
Yes.

Reviewer comment (Evaluation):
Table 4. It's better to show the confidence interval on this table, if possible, since 400 examples is not that large a test set.

Author response:
The primary objective of Table 4 is to underscore the pivotal role of instruction complexity in enhancing coding performance. The table effectively demonstrates that models trained with any round of round 1-4 data (with evolved complexity) outperform round 0 data (without evolved complexity) with a substantial gap. Given the clear difference in performance between any evolved round and non-evolved data, the presentation of confidence intervals is unnecessary to prove that evolved data is better.

Reviewer comment (Evaluation):
Section 6: conclusion. ".. the pivotal role of instruction complexity in enhancing coding performance". Reviewer doesn't think this claim holds well given the analysis in the paper. For example, in Table 4, later rounds (assuming more complexity) lead to a lower performance (or statistically insignificant changes).

Author response:
We appreciate your constructive comments. What we intended to convey is that making instruction more complex benefit code model fine-tuning. To better substantiate our claim and eliminate possible misunderstandings, we would like to further explain Table 4 and the corresponding experimental results. Table 4 demonstrates that models trained with any round of round 1-4 data (incorporating evolved complexity) exhibit significantly better performance than round 0 data (devoid of evolved complexity). Though rounds 1-4 are not a monotonous process with increasing complexity, they are all better than the code-alpaca data with low complexity. This indicates that complexity helps to achieve good performance. Figure 3 further supports this by showcasing that the combination of rounds 0-3 achieves the best performance, indicating the beneficial impact of introducing instruction complexity. We sincerely hope that the clarification could address your concern.

Reviewer comment (Evaluation):
Missing human assessment - It is not clear how useful the final fine-tuned model is outside the benchmarks that focus exclusively on functional correctness. Model hasn't been tested on developer productivity tasks like completion, code refinement.

Author response:
Tasks related to developer productivity can serve as useful assessments of the practical applicability of code models in real-world industrial scenarios. However, even the most advanced LLM, GPT-4, faces challenges when applied to real-life industrial coding tasks. Thus, the most popular evaluation method for code LLMs research is the execution-based function-level code snippet generation, where the generated code snippet/patch undergoes unit tests to ensure functional correctness. We follow this research line to explore methods to improve the performance of function-level code snippet generation. **Our evaluation strategy aligns with the previous and contemporary SOTA works in this research line, such as Starcoder and Codellama, as well as works accepted by ICLR23 and KDD23, including CodeGen and CodeGeeX**. These works similarly exclude human assessment of code. 
    
The chosen evaluation tasks, namely HumanEval, MBPP, DS-1000, and MultiPL-E, collectively cover a spectrum of diverse code completion and insertion scenarios. Specifically, HumanEval addresses function-level code completion, MBPP focuses on task-level code generation, DS-1000 entails statement-level code completion and insertion tasks within the data science domain, and MultiPL-E focuses on diverse programming languages code completion. This comprehensive evaluation framework ensures a thorough examination of the model's utility across various dimensions.

Reviewer comment (Writing):
The paper assumes the reader to be familiar with Evol Instruct, and does not provide sufficient context to follow the method.

Author response:
Thank you for your suggestion. We have included the general Evol-Instruct discussion in the related work section.

Reviewer comment (Evaluation):
Analysis - Complexity and Quantity: This section is not clear to me. What are the results shown in Table 4? How does Table 4 make one conclude that the gains are not due to increase in samples or tokens? Are the 4 rows in Table 4 pass@1 metric for model checkpoints when sequentially trained on each round?

Author response:
In Table 4, all models are fine-tuned separately from scratch, not sequentially. We fine-tune the models with only the specific round data, which contains a similar number of samples (upper part) or tokens (lower part). The results show that with a similar number of samples or tokens, the models trained with evolved data achieve better pass@1. We have updated this section to make it clear.

Reviewer comment (Experiments):
How dependent is the performance of Evol-Instruct on the evolving model? Will the performance substantially improve if we use GPT-4 to generate evolved rounds? Will it worsen if the evolving model is changed to other open sourced alternatives?

Author response:
Please refer to the section "2. Common Response to using different evolution execution models" in the Overall Response. We have provided the results of using GPT-4 or open-sourced models as evolution execution models.

In additional experiments, we replaced GPT3.5 (gpt3.5-turbo) with GPT-4 for generating evolved rounds. HumanEval Pass@1 scores increased from 73.2 to 73.8 for 34B and from 59.8 to 62.2 for 15B.

Furthermore, we explored using the open-source models (OSS) CodeLlama-Instruct-34B for generating evolved instructions. However, it demonstrated relatively low coding performance in response generation. To address this, we fine-tuned it using the code-alpaca dataset and utilized this model for response generation. The OSS-based Code Evol-Instruct tuned models also exhibited performance improvements compared to the code-alpaca fine-tuned model, increasing from 65.9 to 70.1 for 34B and from 45.7 to 55.5 for 15B.

While GPT-4 demonstrates superior coding performance with a score of 88.4 compared to GPT3.5 (gpt3.5-turbo) with a score of 73.2, leveraging a more powerful model for generating evolved rounds does not result in a proportional performance gain (73.8 vs. 73.2). Conversely, despite CodeLlama exhibiting weaker performance than GPT3.5 (gpt3.5-turbo), the performance gap significantly diminishes when utilizing our Code-Evol Instruct to generate evolved rounds (73.2 vs. 70.1). This underscores the observation that employing a stronger model for evolved rounds can somewhat enhance performance, but the extent of improvement is not strongly correlated to the evolution executing model's coding performance. **Our Code Evol-Instruct emerges as the pivotal factor in driving performance enhancement.**

Reviewer comment (Writing):
Section 3.1: We optimised the evolutionary instructions by eliminating deepening and complicating inputs, as well as In-Breadth Evolution. What does this mean? Not clear how 1 and 2 are different, can you describe precisely with an example?

Author response:
Thank you for your reviews. We have updated the section 3.1 to make it more clear.
    
Example:
Original Instruction: `Write a MongoDB query to select all documents in a collection where the field 'category' is 'clothes' and the 'brand' field is not equal to 'Nike'.`

The first method "Add new constraints and requirements..." asks the model to include more requirements in the instruction: `Write a MongoDB query to select all documents in a collection where the field 'category' is 'clothes' and the 'brand' field is not equal to 'Nike', and the 'price' field is greater than or equal to 100 and less than or equal to 500.`

The second method "Replace a commonly used requirement..." asks the model to replace the requirements in the instruction: `Write a MongoDB query to select all documents in a collection where the field 'material' is 'cashmere' and the 'designer' field is not equal to 'Chanel'.`

Reviewer comment (Reproducibility):
Can you provide details of the compute infrastructure involved and hyperparameters in fine-tuning? (number of GPU hours, batch-size, sequence length) and other details)

Author response:
The details of hyper-parameters are included in Section 4.2. We train our models with 24 V100 GPUs, 15 hours for 15B, and 32 hours for 34B.

Reviewer comment (Writing):
The abstract (and some places in the paper) comment on model sizes of Claude and Bard as 'largest closed-source LLMs', this is not verifiable, so I suggest the authors re-word this without commenting on their model size and avoid other such colloquial writing present in the paper to improve its soundness.

Author response:
Thank you for your suggestion. We re-word this without commenting on their model size.