Paper: SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR-2024)

Reviewer comment (Novelty):
The contribution of SWE-Llama is not significant, especially for an AI conference. The paper could better target a software engineering/programming conference.

Author response:
**SWE-Llama is not a significant contribution**: SWE-Llama is not the main contribution (full list in General Response). SWE-Llama is important because it establishes a lower bound for future attempts at the benchmark. It also shows the promise of fine-tuning on SWE-bench-train, which would likely improve performance for models like GPT-4.

Reviewer comment (Experiments):
This method is mainly based on Code Llama while there is no comparison between Code Llama and SWE-Llama.

Author response:
**No comparison between CodeLlama and SWE-Llama**: As discussed in Section 4, we found that open source models like CodeLlama are completely unable to generate well formatted patches. CodeLlama’s performance is 0% in any retrieval setting, and patch generations consistently fail to apply. This prevents any meaningful evaluation of CodeLlama, and is the primary motivation for fine-tuning with SWE-bench-train. We thank you for bringing this up and have clarified this in the new version of our paper.

Reviewer comment (Evaluation):
Some of the experimental analysis is not solid enough. For example, in the “Difficulty correlates with output length” (Section 5), Table 8 only presents all successfully applied patches, and does not show the correlation between difficulty and output length. The length of other patches needs to be taken into account.

Author response:
**Characterizing patches regardless of whether they applied or not**: Thanks for pointing this out. Based on your feedback, we recalculate the metrics (i.e. Total Lines, Added, Removed) for an average patch generation across all patches (not just successfully applied patches). Across all metrics + models, patch generations are much closer in size to gold edits. From this table, it’s clear that models struggle with generating longer patches that are correctly formatted. Further inspection of these flawed generations, as shown in Appendix F, indicate that hallucinations, abiding to existing code style/structure, and referencing long range dependencies correctly are common formatting errors that surface more frequently in longer generations.

| Model | Total Lines | Added | Removed | Functions | Files |
| --- | --- | --- | --- | --- | --- |
| Claude 2 | 27.2 | 6.6 | 3.3 | 1.2 | 1.1 |
| Claude 2 (Gold) | 61.6 | 17.8 | 8.6 | 2.6 | 1.4 |
| ChatGPT-3.5 | 42.0 | 6.1 | 3.9 | 1.7 | 1.0 |
| ChatGPT-3.5 Gold | 44.5 | 12.7 | 5.5 | 2.1 | 1.2 |
| GPT-4 | 22.4 | 4.4 | 1.8 | 0.8 | 0.9 |
| GPT-4 Gold | 50.3 | 14.0 | 6.5 | 2.3 | 1.3 |
| SWE-Llama 13b | 68.9 | 9.5 | 4.3 | 2.5 | 1.6 |
| SWE-Llama 13b Gold | 61.5 | 17.8 | 8.6 | 2.6 | 1.4 |
| SWE-Llama 7b | 78.9 | 10.1 | 7.6 | 2.5 | 1.5 |
| SWE-Llama 7b Gold | 65.1 | 18.8 | 9.0 | 2.7 | 1.5 |

Reviewer comment (Experiments):
As the experimental results of GPT-4 are on a 20% random subset of SWE-bench while there is no comparison of other models on the same subset. If we only look at this part of the subset, are all the conclusions in the paper still valid/consistent?

Author response:
**Performance on 25% Subset**: Based on your comment, we recompute the main evaluation results (% Resolved, % Apply metrics for BM25, Oracle retrieval settings) for the 25% subset of SWE-bench that GPT-4 was evaluated on to check if the results might be different. The performance here is consistent with the main results and conclusions already in the paper. This is expected because the selection procedure for the subset is random. We have added a discussion of this in the paper (Appendix C.2).

| Model | BM25 Retrieval  | | "Oracle" Retrieval | |
|---|---|---|---|---|
|| **% Resolved** | **% Apply** | **% Resolved** | **% Apply** |
| Claude 2    | 2.26 $\uparrow$ 0.30   | 28.57 $\downarrow$ 1.29 | 4.88 $\uparrow$ 0.08| 49.36 $\uparrow$ 2.37 |
| ChatGPT-3.5 | 0.17 $\downarrow$ 0.03 | 8.04 $\downarrow$ 2.46 | 0.84 $\uparrow$ 0.32| 11.67 $\downarrow$ 0.71 |
| GPT-4 | 0.00 $-$ 0.00| 4.50 $-$ 0.00    | 1.74 $-$ 0.00 | 13.24 $-$ 0.00 |
| SWE-Llama 7b| 0.42 $\downarrow$ 0.28 | 37.50 $\downarrow$ 0.34 | 2.12 $\downarrow$ 0.89 | 51.56 $\downarrow$ 3.24 |
| SWE-Llama 13b   | 0.70 $-$ 0.00| 40.47 $\uparrow$ 1.06 | 4.36 $\uparrow$ 0.39| 49.13 $\downarrow$ 3.01 |

Reviewer comment (Novelty):
Although benchmark and LLM evaluation on it are valuable, the paper does not present any novel solutions to the task in the benchmark. This limits the contribution.

Author response:
1. **Lack of novel solutions to the task in the benchmark**: Past precedents such as the GLUE benchmark for NLU, WMT benchmarks for machine translation, and WikiText103 for LM perplexity, have shown the importance of simply presenting a tough benchmark for driving progress. We believe that presenting work which focuses primarily on introducing a challenging, useful benchmark can be very useful for the ML and NLP communities. Through experiments and analysis we also present actionable insights for future improvement (See Section 5, Appendix C/D/F).

Reviewer comment (Presentation):
Please reorganize the paper so tables and figures are collocated with the text. Currently, it is hard to read when tables are referenced out of order and explained very far from their location in the paper.

Author response:
2. **Reorganization of the tables/figures to be closer with text**: In the latest upload of our paper, we have incorporated this suggestion and put figures closer to text and made sure tables are mentioned in order. This should improve the readability. Thanks for this suggestion.

Reviewer comment (Writing):
This sentence, especially its last part, is unclear: "We compare the BM25 retrieval results against the oracle retrieval setting in Table 3, where we see that BM25 retrieves a superset of the oracle files in about 40% of instances with the 27,000 token context limit but only also excludes all of the oracle files in over half of instances.". I think this is trying to explain the results in Table 3 and trying to say that in around half of cases BM25 does not retrieve any of the oracle files. Is this what you are trying to say? Please explain or rephrase.

Author response:
3. **Unclear sentence discussing retrieval**: Yes, your understanding is correct. To make the paper clearer we’ll be replacing the mentioned sentence with the following: "We compare the BM25 retrieval results with those of the 'oracle' retrieval setting, as shown in Table 3. We observe that in approximately 40% of instances, BM25 retrieves a superset of the oracle files for the 27000-token context limit. However, in almost half of the instances with the 27000-token limit, it retrieves none of the files from the 'oracle' context."

Reviewer comment (Evaluation):
It seems that none of the models is doing well when the benchmark is used.

Author response:
3. **Results are hard to interpret/distinguish between models**: We find that many of the challenges to solving long-context code problems are common to all models evaluated. While our results show that some models have better average success, overall performance on SWE-bench is so low across the board that it can be difficult to differentiate between models’ behavior. We think that SWE-bench can provide an important benchmark in guiding LM progress on the continual refinement of long context understanding, high and low-level reasoning, and grounded generation.

Reviewer comment (Evaluation):
It would be nice if the benchmark can be used to more clearly indicate where the problem in the language model lies.

Author response:
1. **More in-depth and insightful analysis**: Given the number and diversity of problems in SWE-bench, insightful analysis of generated solutions is certainly a challenge. To provide better insight, we’ve expanded our qualitative analysis in the paper to analyze 11 full generations from both Claude and SWE-Llama, showing that:  
Models tend to propose shortcut solutions which do not consider all possible inputs to the code.  
Models have a simplistic coding style that ignores the style and utilities of the overall codebase, leading to less accurate and portable solutions.  
We provide more insights and possible solutions, e.g., prompting with documentation, learning from interaction, and test generation, for solving these problems in the paper.

Reviewer comment (Writing):
It would be nice if the exact contributions of the paper are stated more clearly.

Author response:
2. **Our exact contributions** — We restate our exact contributions in the general response comment and clarify them in the paper. Please refer to the general response for more information on this concern.

Reviewer comment (Evaluation):
In section 1, the authors point out that there is a need for a challenging benchmark that can be used to check the abilities of language models. Although the results have been reported, it is unclear how far they evaluate the specific abilities or weaknesses. The results are general, and seem to apply to all the models without discerning the strengths/abilities or weaknesses of a particular model.

Author response:
3. **Results are hard to interpret/distinguish between models**: We find that many of the challenges to solving long-context code problems are common to all models evaluated. While our results show that some models have better average success, overall performance on SWE-bench is so low across the board that it can be difficult to differentiate between models’ behavior. We think that SWE-bench can provide an important benchmark in guiding LM progress on the continual refinement of long context understanding, high and low-level reasoning, and grounded generation.

Reviewer comment (Experiments):
At this stage, since all the models are performing poorly, perhaps there is a need for a benchmark that is neither too simple, but not as general as SWE-bench, to allow some aspects of the models to be better tested and reported.

Author response:
4. **Is there a need for a benchmark even simpler than SWE-bench?**: LM progress moves fast and evaluation benchmarks are quickly outdated. We believe that setting a bar as high as SWE-bench is of great interest to the community since it could serve as a more persistent aspirational achievement for LMs. Achieving high accuracy on SWE-bench would be both incredibly useful to actual software engineers and also indicate substantial technical progress in the development of LMs.  
Additionally, provided the structure of SWE-bench, it is possible for users to evaluate their models in slightly simpler settings (e.g. with the Oracle and Oracle-collapsed baseline), which can provide researchers some improved insights in the development of their own models.