Paper: Learning Performance-Improving Code Edits (ICLR-2024)

Reviewer comment (Experiments):
Regarding the experiment in Section 4.1: In Table 2, the performance of GPT-4 under the Few-Shot and CoT methods for code optimization was not tested.

Author response:
### **[R4-Q3] The Performance of GPT-4 under Few-Shot and CoT were not tested**

At the time of writing, we did not have the funds to run all experiments with GPT-4. We were able to get the funds, and we have evaluated the outputs, and we will include them in the draft. For GPT4, both Few Shot and CoT prompting improve over Instruction prompting but underperform retrieval-based prompting. For example, Best@4 for CoT has a %Opt of 38.60% and a speedup of 1.51$\times$, while Few-Shot has a %Opt of 30.83% and a speedup of 1.37$\times$.


| Prompting Method | %Opt | Speedup |
|------------------|--------|---------|
| Instruction Only (Best@4) | 16.58% | 1.23$\times$|
| CoT (Best@4) | 38.60% | 1.51$\times$ |
| Few-Shot (Best@4) | 30.83% | 1.37$\times$ |
| Dynamic Retrieval (Best@4) | 69.03% | 3.56$\times$|

Reviewer comment (Experiments):
Different types of code have varying scopes and complexities for optimization. The paper does not clearly state the proportion of each language type in the training data nor in the test code.

Author response:
### **[R4-Q4] Different types of code have varying scopes and complexities for optimization. The paper does not clearly state the proportion of each language type in the training data nor in the test code.**

Our training/validation/test sets consist entirely of C++ programs. We focus on C++ since it is a performance-oriented language compatible with the gem5 simulator. We will update the draft to make this more clear.

Reviewer comment (Theory):
How do you view the impact of training LLMs using synthetic data (generated either by the trained model itself or from other models)? Assuming synthetic data is generated by the trained model M and undergoes data filtering, how would the output distribution of M change?

Author response:
### **[R4-Q1] How do you view the impact of training LLMs using synthetic data?**

The last three rows of fine-tuning results in Table 3 show the result of fine-tuning on synthetic data generated by GPT3.5 (in addition to data from PIE); we see increased generalization. The synthetic data was generated by the pre-trained GPT3.5 turbo model, and the optimized programs were then generated by GPT3.5 fine-tuned on 4K high-quality pairs. GPT3.5-turbo fine-tuned on the PIE dataset. Generally, training with this synthetic data enhances generalization ability. Please let us know if this does not answer your question, and we will be happy to provide additional clarifications.

Reviewer comment (Writing):
I noticed a statement regarding the comparison of LoRA training: "We hypothesize that this gap may be because performance optimization examples do not occur naturally in the training data." Can you provide a clearer version of this statement, or offer a more specific explanation?

Author response:
### **[R4-Q2] Can you provide a clearer version of the statement regarding LoRA?**

We hypothesize that this gap may be because performance optimization examples do not occur naturally in the training data.

Recent work has shown that the effectiveness of parameter-efficient methods depends on the training data. For example, [1] find that "PEFT techniques are slower to converge than full tuning in low/medium-resource scenarios," and [2] find that LoRA is least effective for challenging tasks like mathematical reasoning. Together, these works indicate that the performance of PEFT may be heavily task-dependent. Our hypothesis is based on the fact that LoRA only changes a small subset of the model's parameters, and is likely most helpful when the base model has some proficiency for the task (due to pre-training), and LoRA can help "adapt" the model of the task further. Given that LLMs generally struggled in program optimization without retrieval or full fine-tuning, we hypothesize that the challenging nature of the problem and a potential lack of pre-trained proficiency pose challenges for LoRA. We will add these new citations in the next version and clarify this point.
[1] He, Junxian, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. "Towards a unified view of parameter-efficient transfer learning." arXiv preprint arXiv:2110.04366 (2021).
[2] Fine-Tuning LLMs: LoRA or Full-Parameter? An in-depth Analysis with Llama 2

Reviewer comment (Presentation):
In appendix A.1, Figure 3 contains two programs that are functionally different.

Author response:
> ### **[R3-Q7] “Figure 3 of A.1 contains two programs that are functionally different”**

Problem URL: https://atcoder.jp/contests/abc127/tasks/abc127_c

Input Space: $$1 <= N <= 1e5; 1 <= M <= 1e5; 1 <= L_i <= R_i <= N$$

In addition to ~100 unit tests, we randomly generated 10,000 tests within the provided input bounds, and found that the two programs produced equal outputs on all of these inputs. Both programs aim to find the range between the maximum of the first set of elements and the minimum of the second set of elements. If the maximum of the first elements is greater than or equal to the minimum of the second elements, they both output 0. Otherwise, they output the difference plus 1. The first program sorts two vectors containing all $a$s and $b$s to find these values, while the second program directly computes the maximum $a$ and minimum $b$ during the input phase. However, the second program is more efficient, as it operates in linear time, avoiding the sorting overhead present in the first program.

Reviewer comment (Presentation):
In appendix A.1, Figure 4(a) is so bad that it seems unlikely in a C++ competition.

Author response:
> ### **[R3-Q10] The submission in Figure 4(a) in A.1 seems unlikely from a C++ competition**

The submissions are all from CodeNet (Puri et al., 2021). The dataset obtained from the [AtCoder](https://atcoder.jp/) or [AIZU](https://judge.u-aizu.ac.jp/) websites, and DeepMind’s [AlphaCode paper](https://arxiv.org/pdf/2203.07814.pdf) refers to the CodeNet dataset as a “competitive programming dataset.” This dataset deliberately contains a wide variety of different submissions, including early drafts that coders improved over time, as well as submissions from both beginners and experts. This diversity makes the dataset useful as a performance optimization benchmark, since we can examine how to help beginners and experienced programmers improve the performance of their code.

For reference, here is the URL to the problem in Figure 4: [B - One Clue](https://atcoder.jp/contests/abc137/tasks/abc137_b)

Here are some other URLs to problems in the PIE test set: 

- [F - Fractal Shortest Path](https://atcoder.jp/contests/panasonic2020/tasks/panasonic2020_f)

- [E - Jigsaw](https://atcoder.jp/contests/agc017/tasks/agc017_e)

- [Problem B: Compress Files](https://judge.u-aizu.ac.jp/onlinejudge/description.jsp?id=2080)

Reviewer comment (Evaluation):
In appendix A.1, Figure 5 contains two programs that are functionally different and also have different interfaces for execution, making it unclear how it passes the correctness test.

Author response:
> ### **[R3-Q8] “Figure 5 contains two programs that are functionally different and also have different interfaces for execution (unclear how it passes correctness test)”**

Problem URL: https://atcoder.jp/contests/abc055/tasks/abc055_b

Input Space: $$1 <= N <= 1e5$$

After brute-force enumerating over all possible inputs in the specified input range, both programs are equivalent. Both programs calculate the factorial of an integer $n$ modulo $10^9 + 7$, but the second program is significantly faster, as it eliminates the overhead of precalculation and array storage, resulting in a faster execution for each individual input of $n$.

Reviewer comment (Presentation):
In appendix A.1, Figure 6 contains two programs that are functionally different, and Figure 6(b) is missing an obvious break statement that would help performance.

Author response:
> ### **[R3-Q9] Figure 6 contains two programs that are functionally different** 

Problem URL: https://atcoder.jp/contests/abc060/tasks/abc060_b

Input Space: $$1<= A <= 100; 1 <= B <= 100; 0 <= C < B$$

After brute-force enumerating over all possible inputs in the specified input range, both programs were equivalent. Both programs determine if an integer solution exists for the equation $(A \times i) - (B \times j) = C$, but the second program is significantly more efficient by leveraging modular arithmetic, reducing the problem to a single loop with a maximum of $B$ iterations, compared to the first program's exhaustive search with nested loops.

---

> ### **[R3-Q11] Fig 6b is missing an obvious break statement to help performance**

Thanks for pointing out the additional opportunity for optimization. While the models are trained to generate faster programs, there may still be room for improvement. We will clarify this point in the caption that: `the example on the right is faster, but the generated code could have been even faster if it included a break statement.`

Reviewer comment (Experiments):
A random check of some entries in the supplementary material shows they are consistent with examples in appendix A.1 and are not plausible for competitive C++ programming submissions.

Author response:
> ### **[R3-Q6] Examples in the appendix are “functionally different”**

All three of these functions pass all of our unit tests described in Section 2. Moreover, we exhaustively tested all three based on the accepted range of inputs from the problem description: all three program pairs are equivalent. 

The problems in our benchmark come with input ranges, and programs are only required to provide the correct output on inputs inside these ranges; we will clarify this point in our paper. We give details below; we are also happy to share the scripts that we used to exhaustively test the problems.

Regarding the broader issue of how we check correctness, we emphasize that the use of unit tests to check correctness is the standard way to check correctness in competitive programming, and is also standard in the program synthesis literature (under the name “programming by example”); indeed, example-based synthesis has been integrated into commercial tools such as the FlashFill tool for Excel [1]. We would also like to note that CodeNet by default comes with only 5 test cases per program, and we increased the coverage to a median of over 100 test cases per program by augmenting additional test cases from Alphacode (Section 2).

[1] Sumit Gulwani, [Automating String Processing in Spreadsheets using Input-Output Examples](https://dl.acm.org/doi/abs/10.1145/1925844.1926423?casa_token=_p-wn3-qwpwAAAAA:lMSWSPoREj_71zl3Di3vuq79-eY67y2F-tEzu1L72gXWmtWfjw07lGkVgrBw2mKbXHRtCoInN79IUw). In POPL, 2011.

Reviewer comment (Evaluation):
The evaluation setup, as stated on page 6, is such that for generations that are either incorrect or slower than the original program, a speedup of 1.0 is used, meaning only good generations are considered in calculating the metric. This partially explains the 6.86X result.

Author response:
> ### **[R3-Q12] "The evaluation setup is such that only good generations are considered in calculating the metric.**

This strategy reflects the natural way a program optimization tool would be used: keep the best program collected, including the initial program. In particular, if a programmer realizes the generated program is incorrect or slower, they can simply revert to the original and have a speedup of 1.0. This approach aligns with standard software development practices, where unproductive changes are discarded. In fact, for GPT-3.5 with self-play (SP), this strategy actually has less impact than other models because so many programs are optimized: GPT-3.5 (SP) improved performance for 87.68% of programs in the benchmark by at least 10%. Thus, the strategy was only applied to a small fraction of programs (at most 12.32%).

Reviewer comment (Evaluation):
The rationale for using gem5 simulation instead of measuring wall time makes sense, but it risks being overly focused on a single CPU (Intel Skylake). Furthermore, gem5 simulation does not capture big O complexity, which is often a goal in optimizing code.

Author response:
> ### **[R3-Q1] Will using gem5 risk being overly-focused on a single CPU (Intel Skylake)?**

In Section 4, we analyzed 120 randomly selected programs in the test set, and all optimizations seemed generic with nothing specific to x86, let alone Skylake. Generally all optimizations we observed would execute faster on any backend: for example, using a faster sorting algorithm will almost always be faster on any backend. 

We went further to check this systematically. To our knowledge the most plausible ways to customize C++ for a certain backend is through direct writing of assembly, SIMD/vector programming, and pragma statements (e.g. to specify memory prefetching). For the GPT3.5 model trained with self-play, **none** of the solutions contained assembly or vector programing, and only 0.3% of optimized generations contained pragmas that could affect execution time. It is intuitive that the model doesn’t learn this behavior, because programers doing program competitions seem to rarely focus on optimizing for the program website backend. 

---

> ### **[R3-Q2] Also gem5 simulation still does not capture big O complexity which is often a goal in optimizing code.**

Big O complexities are impossible to automatically infer in the general case, and thus cannot be the basis for an automatic metric. Furthermore, Big O captures *asymptotic* complexities and can hide large constants. By measuring actual runtime on provided test cases, our evaluation best aligns with how user submissions are evaluated in practice.

Reviewer comment (Experiments):
It is counter-intuitive that Section 3.2 only uses 4K out of the 77K pairs in PIE to fine-tune GPT-3.5, then creates and uses 3K synthetic data. The rationale for these choices is unclear and seems to contradict the quality claim of the PIE dataset.

Author response:
> ### **[R3-Q3] Why do we only use 4K programs in section 3.2 and add more synthetic programs later, instead of real programs.**

As described in Section 3.2, the reason we did not finetune GPT-3.5-turbo on our full PIE dataset is due to our limited API budget. At a high level, our results in Table 3 demonstrate the following for the CodeLlama models:

- Performance-conditioned models trained on PIE perform the best (by a wide margin)
- Models trained on the 4K "high quality" subset of PIE perform second best
- Models trained on all of PIE (without performance conditioning) are third

These results make sense, because finetuning on all of PIE without performance conditioning encourages the model to generate modest speedups. We ideally would have finetuned GPT-3.5-turbo using performance conditioning on the full PIE dataset, but we did not have the budget to do so. Thus, we used the second best option, which was to train GPT-3.5-turbo on the high quality examples, which was also significantly cheaper (~20x cheaper).

The synthetic data was introduced to solve a separate issue, which is the limited number of problems in the dataset. Even if we have a large number of submissions, they are all for the same 1474 problems. Our synthetic data consists of *new problems*, which helps our techniques generalize better to new problems (recall that our train/test split is at the level of problems, not code submissions). Because of this, synthetic data helps improve performance compared to just using PIE. On the page 8 footnote we report an ablation demonstrating the gains are from the type of data added, not the quantity of data.

Reviewer comment (Evaluation):
The study mostly focuses on the percentage of performance, and in cases where the runtime of the generated program is slower or the code is incorrect, speed up is considered one. This is not a good approach. In particular, a study is needed on what category of programs LLMs often fail to produce correct code or optimized code. In such a study, it will be easier to infer in which cases it is better not to use the LLMs. Or which category of programs needs to be further improved.

Author response:
Thank you for the suggestion to investigate error analysis!
We performed this analysis for GPT-3.5 fine-tuned with synthetic data. Specifically, we analyzed the generated programs that PIE fails to optimize and the root cause of each failure. Our findings are summarized below:
| Result | Percentage |
|-|-|   
| Failed to compile (syntax/type errors) | 12.51% |
| Compiled, but got [95-100%] of test cases wrong | 27.59% |
| Compiled, but got (75, 95%] of test cases wrong | 12.08% |
| Compiled, but got (25, 75%] of test cases wrong | 10.84% | 
| Compiled, but got (0-25%] of test cases wrong (at least 1 test case was wrong) | 6.90% |
| Ran all test cases, but the program was slower than the original | 9.93% |
| Ran all test cases, but the program was the same speed as the original | 9.40% |
| Ran all test cases, the program was faster, but not 1.1$\times$ speedup or higher | 10.75% |


In summary, a large fraction (~60%) of the failures happen because the proposed changes break a unit test. In about 30% of the cases, the model produces a correct, but the generated program is either slower (10%) or doesn't meet our threshold for speedup (10%). Finally, in about 10% of the cases, the generated program has a syntax error. Additionally, with test cases, we find that when the model gets the program wrong, it seems to most often get it "quite wrong" by missing most test cases. We will update the existing analysis (Appendix A) with these new findings.

Finally, we conduct additional analysis to investigate the properties of programs that PIE fails to optimize. The results show a mild negative correlation between the problem description length and average accuracy (-0.15) and between the source program length and average accuracy (-0.26), suggesting longer inputs slightly reduce accuracy. Additionally, the average speedup has a mild negative correlation with both the problem description length (-0.16) and the source program length (-0.11), indicating a minimal impact of length on speedup compared to correctness. Overall, this analysis reveals that language models struggle to generate a correct program when faced with larger source programs and challenging problems, but their ability to optimize programs is minimally impacted. This motivates a future work direction where techniques from program repair may be combined with PIE for better results.

Reviewer comment (Experiments):
For synthetic data, it is mentioned that semantic duplicates are tracked, but no information is provided on how semantic similarity is conducted. Is it also based on CodeBertScore?

Author response:
In Section A.4 of the Appendix, we describe how we check semantic similarity. In detail, because our goal is to only demonstrate inequivalence, not equivalence, we use a set of test cases, and ensure that at least 1 output is different from programs in the PIE dataset. We use 200 test cases for all programs, and the generated program must run without errors on at least 20 of these. To achieve this, we had to execute more than 1.4 million binary input pairs and track all the outputs.

Reviewer comment (Evaluation):
It is interesting to see that Perf-Cond has decreased the percentage of correctness. Any reason why Perf-Cond degrades the ability of the LLMs to produce the correct code?

Author response:
We believe that conditioning the model only to generate programs with a 10/10 optimization rate may constrain the number of optimizations available for any given input. To verify this, we experimented using the first 6 generations from the 7b Perf-Cond model when conditioned on 10/10 versus combining the first 2 generations when conditioned on 10/10, 9/10, and 8/10. When we did this, we saw a %Correct increase from 59.95% to 64.36%. These figures support the explanation that performance labels may restrict which generated programs are correct.

Reviewer comment (Presentation):
The presentation could be improved as an example, Table 1 is not referenced anywhere in the text, or in Table 2, it is better to group the Dynamic Retrievals together. The additional horizontal line is confusing.

Author response:
Thank you for pointing these presentation details out to us; we will update the draft based on your feedback!

Reviewer comment (Experiments):
In Table 2, why is the choice of models not consistent across different methods? Like, there is no few shots learning or COT for GPT-4.

Author response:
At the time of writing, we did not have the funds to run all experiments with GPT-4. We were able to get the funds, and we have evaluated the outputs, and we will include them in the draft. For GPT4, both Few Shot and CoT prompting improve over Instruction prompting but underperform retrieval-based prompting. For example, Best@4 for CoT has a %Opt of 38.60% and a speedup of 1.51$\times$, while Few-Shot has a %Opt of 30.83% and a speedup of 1.37$\times$.
| Prompting Method | %Opt | Speedup |
|------------------|--------|---------|
| Instruction Only (Best@4) | 16.58% | 1.23$\times$ |
| CoT (Best@4) | 38.60% | 1.51$\times$ |
| Few-Shot (Best@4) | 30.83% | 1.37$\times$ |
| Dynamic Retrieval (Best@4) | 69.03% | 3.56$\times$ |

Reviewer comment (Experiments):
In Table 2, why is the choice of models inconsistent across different methods? Like, there are no few shot learning or COT for GPT-4.

Author response:
At the time of writing, we did not have the funds to run all experiments with GPT-4. We were able to get the funds, and we have evaluated the outputs, and we will include them in the draft. For GPT4, both Few Shot and CoT prompting improve over Instruction prompting but underperform retrieval-based prompting. For example, Best@4 for CoT has a %Opt of 38.60% and a speedup of 1.51$\times$, while Few-Shot has a %Opt of 30.83% and a speedup of 1.37$\times$.
| Prompting Method | %Opt | Speedup |
|------------------|--------|---------|
| Instruction Only (Best@4) | 16.58% | 1.23$\times$ |
| CoT (Best@4) | 38.60% | 1.51$\times$ |
| Few-Shot (Best@4) | 30.83% | 1.37$\times$ |
| Dynamic Retrieval (Best@4) | 69.03% | 3.56$\times$ |

Reviewer comment (Evaluation):
Why does Perf-Cond degrade the ability of the LLMs to produce the correct code?

Author response:
We believe that conditioning the model only to generate programs with a 10/10 optimization rate may constrain the number of optimizations available for any given input. To verify this, we experimented using the first 6 generations from the 7b Perf-Cond model when conditioned on 10/10 versus combining the first 2 generations when conditioned on 10/10, 9/10, and 8/10. When we did this, we saw a %Correct increase from 59.95% to 64.36%. These figures support the explanation that performance labels may restrict which generated programs are correct.

Reviewer comment (Reproducibility):
How semantic similarity is conducted where semantic duplicates are tracked?

Author response:
In Section A.4 of the Appendix, we describe how we check semantic similarity. In detail, because our goal is to only demonstrate inequivalence, not equivalence, we use a set of test cases, and ensure that at least 1 output is different from programs in the PIE dataset. We use 200 test cases for all programs, and the generated program must run without errors on at least 20 of these. To achieve this, we had to execute more than 1.4 million binary input pairs and track all the outputs.