Paper: RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (ICLR-2024)

Reviewer comment (Novelty):
To me, the novel research contributions of the paper are a bit limited, especially for an AI conference. The paper could better fit a software engineering/programming conference. The significance of the work could be more clearly stated.

Author response:
> The novel research contributions of the paper are a bit limited, especially for an AI conference.

We would like to highlight the significance of our work:

1. We hope that RepoBench can address the evaluation of code generation in LLMs at current stage. This is a key area in the field of AI, as understanding and improving code generation capabilities are vital.
2. RepoBench-C can be used to facilitate tasks involving long-context generation. The customizable token number feature of RepoBench-C allows the construction of 2k, 8k, 16k and even 128k versions, making it a benchmark for evaluating the efficacy of LLMs in handling long context length. This flexibility is particularly important in the current landscape of AI research, where handling longer contexts is becoming increasingly crucial.
3. We are encouraged by the interest RepoBench has already garnered in the LLM community. The StarCoder team has noticed the performance of StarCoder here, leading to a collaboration study focused on repo-level model training for next version. You can check the preliminary ablation study we already have shown in the global response.

Reviewer comment (Experiments):
To evaluate RepoBench-R, the authors selected three baseline strategies for the retrieval task, namely, random retrieval, lexical retrieval, and semantic retrieval. The selection of baseline strategies for RepoBench-R, particularly the inclusion of random retrieval and lexical retrieval, are weak baselines, which may not effectively demonstrate the distinctive capabilities of the proposed benchmark. In that sense, the results presented in Section 4.1 are under expectation and I think that previous benchmarks may also demonstrate the ability of these strategies. A more competitive baseline selection including LLMs would enhance the work.

Author response:
>  Weak baselines selection of RepoBench-R.

First, we would like to emphasize the relevance of Jaccard similarity as a baseline. It has been revealed through reverse engineering studies online that GitHub Copilot uses Jaccard similarity backend for retrieval.

In our initial experiments, we evaluated both encoder-only and decoder models, such as CodeGPT and CodeGen. While encoder-only models demonstrated superior performance for retrieval tasks, we initially omitted these results due to page constraints. In response to your feedback, we plan to include these findings to present a more comprehensive evaluation.

We are also excited to share that we have conducted additional experiments with more recent encoding models, specifically CodeT5+ and Intruct-XL. The results from these experiments have shown **Instruct-XL demonstrates similar performance to UnixCoder**.

Table 4: Results of CodeGPT, CodeGen, CodeT5+ and Instruct-XL as retriever on RepoBench-R.

|       Language |  Model     | acc@1 (Easy XF-F) | acc@3 (Easy XF-F) | acc@1 (Easy XF-R) | acc@3 (Easy XF-R) | acc@1 (Hard XF-F) | acc@3 (Hard XF-F) | acc@5 (Hard XF-F) | acc@1 (Hard XF-R) | acc@3 (Hard XF-R) | acc@5 (Hard XF-R) |
|------------------|-----------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|
| **Python**       |     CodeGPT       | 15.86 | 47.12 | 16.83 | 46.92 | 8.46 | 23.15 | 36.71| 7.90 | 22.07 | 35.62 |
| | CodeGen | 20.15 | 53.90 | 22.78 | 55.02 | 12.81 | 30.89 | 45.96| 13.98 | 33.15 | 46.73 |
| | CodeT5+ | 17.86 | 50.61 | 19.25 | 51.63 | 8.39 | 22.53 | 36.02| 8.97 | 24.02 | 36.68 |
| | Instruct-XL | 26.37 | 61.64 | 31.92 | 65.03 | 18.12 | 39.12 | 52.62| 21.07 | 41.50 | 55.38 |
| **Java**       |     CodeGPT       | 15.89 | 48.09 | 17.60 | 49.20 | 7.81 | 23.31 | 37.59| 7.98 | 22.30 | 37.40 |
| | CodeGen | 18.49 | 51.84 | 23.28 | 54.12 | 9.99 | 28.30 | 43.06| 13.30 | 31.35 | 46.55 |
| | CodeT5+ | 15.73 | 46.75 | 16.90 | 49.52 | 6.22 | 17.93 | 29.66| 6.95 | 19.65 | 32.17 |
| | Instruct-XL | 16.88 | 50.58 | 26.05 | 60.57 | 10.69 | 28.09 | 41.58| 17.83 | 37.67 | 50.40 |

Reviewer comment (Evaluation):
The metrics used for code completion, i.e., EM and Edit Similarity, are unusual. The authors could consider more widely used metrics such as pass@k and CodeBLEU?

Author response:
> Why not using widely used metrics such as pass@k and CodeBLEU?

Firstly, our selection of EM and ES metrics simply aligns with  prior research on code completion. For instance, [CodeXGLUE](https://arxiv.org/pdf/2102.04664.pdf), [LongCoder](https://arxiv.org/abs/2306.14893). 

Regarding pass@k, we acknowledge its potential as an ideal evaluation metric due to its emphasis on executable code. However, our strategy prioritizes extensive data utilization to accurately reflect the capabilities of models across thousands of codebases. Implementing pass@k involves ensuring the executability of each piece of code, which is a complex task given the diverse environmental configurations and dependency issues common in library code. The logistical challenges associated with verifying the executability of such a vast array of code samples render pass@k less feasible for our study's scope. Nonetheless, we agree that pass@k could be a valuable addition in a more controlled or smaller-scale setting and will consider it for future research where applicable.

As for CodeBLEU, we appreciate your suggestion and have taken steps to incorporate it into our evaluation. Our latest experiments, as detailed in the Global Response, already include CodeBLEU as an additional metric. This inclusion should provide a more rounded evaluation of the models’ capabilities.

Despite these challenges, we believe that EM, ES and CodeBLEU are effective in reflecting the performance of models on dataset with a large scale.

Reviewer comment (Evaluation):
Why not using widely used metrics such as pass@k and CodeBLEU?

Author response:
> Why not using widely used metrics such as pass@k and CodeBLEU?

Firstly, our selection of EM and ES metrics simply aligns with  prior research on code completion. For instance, [CodeXGLUE](https://arxiv.org/pdf/2102.04664.pdf), [LongCoder](https://arxiv.org/abs/2306.14893). 

Regarding pass@k, we acknowledge its potential as an ideal evaluation metric due to its emphasis on executable code. However, our strategy prioritizes extensive data utilization to accurately reflect the capabilities of models across thousands of codebases. Implementing pass@k involves ensuring the executability of each piece of code, which is a complex task given the diverse environmental configurations and dependency issues common in library code. The logistical challenges associated with verifying the executability of such a vast array of code samples render pass@k less feasible for our study's scope. Nonetheless, we agree that pass@k could be a valuable addition in a more controlled or smaller-scale setting and will consider it for future research where applicable.

As for CodeBLEU, we appreciate your suggestion and have taken steps to incorporate it into our evaluation. Our latest experiments, as detailed in the Global Response, already include CodeBLEU as an additional metric. This inclusion should provide a more rounded evaluation of the models’ capabilities.

Despite these challenges, we believe that EM, ES and CodeBLEU are effective in reflecting the performance of models on dataset with a large scale.

Reviewer comment (Experiments):
How is the proposed benchmark compared to previous benchmarks for code completion?

Author response:
> The paper lacks a comprehensive comparison with previous benchmarks about code completion.

Thank you for highlighting the need for a more thorough comparison with previous benchmarks. While our paper does include some discussion on 'code completion datasets' in related work section (Section 2), we acknowledge that this discussion might be somewhat limited, and we appreciate your guidance on enhancing this aspect of our work.

Firstly, as you pointed out, RepoBench distinguishes itself by focusing on repository-level tasks, which we believe is not only practical but also closely aligns with real-world applications. Also, we focus on using the most recent repositories to mitigate the issue of data leakage. In contrast to benchmarks that might focus on narrower datasets, RepoBench encompasses thousands of repositories. This extensive coverage is critical in evaluating modern models, especially considering that some may employ strategies like 'rephrasing' to achieve high scores on benchmarks like HumanEval. Our broad dataset scope ensures a more thorough and diverse evaluation.

Unlike previous retrieval tasks that often focus on simple semantic matching, our benchmark's code retrieval is contextualized within the code completion process. This adds a layer of practical significance, especially for projects aimed at enhancing code completion tools. 

In light of your valuable suggestion, we plan to include a more detailed comparison of both retrieval and completion aspects in our revised paper.

Reviewer comment (Novelty):
It is not clear what new insights RepoBench and experiments on it contribute to the field. Were the results previously unknown or unexpected?

Author response:
We understand your concern regarding the novel insights provided by RepoBench. Our primary contribution through RepoBench is in addressing the need for benchmarks that evaluate repository-level code auto-completion, a gap that exists in current benchmarks focused mostly on single-file tasks. This focus is particularly relevant given that real-world applications of code models often require a comprehensive repository-level approach for optimal functionality.

Additionally, RepoBench has the ability to construct generation tasks with customizable context lengths (e.g., 2k, 8k, 16k, 64k, etc.). This flexibility allows for an in-depth analysis of model performance under varying long-context scenarios


During our experiments, we did encounter several intriguing findings that were unexpected and might be insightful:

1. **Multilingual Code Generation**: While current models show promising results in Python, matching Codex in some cases, we observed a noticeable performance gap in Java code generation
2. **Long Context Generation**: On repobench-c with 8k max length, StarCoder is still trailing behind the capabilities of Codex, highlighting a crucial area for future development.
3. **Repo-Level performance**: Our initial assumption was that repo-level code generation would be generalizable. However, we found that models like StarCoder did not perform well at this level, suggesting they might need to be trained with repository-level data.  This insight led to a collaborative effort with the StarCoder team and our subsequent contribution to the model improvement. We show some of the preliminary ablation study in a global response, which demonstrates that repository-level training can improve the performance dramatically.
4. **Retrieve as much as possible**: Merely retrieving the gold code snippet is not always the most effective approach. Instead, filling the context with as much 'relevant' code as possible from the repository tends to improve performance.
5. **Positioning of Cross-File Context**: Unlike long context generation in textual documents where models pay more attention to the beginning of the prompt, in repository-level code generation we found that placing more relevant cross-file snippets towards the end of the prompt (closer to the in-file context) can improve performance.

Reviewer comment (Experiments):
This might not be a weakness of the paper per se, but it concerns me a bit that random retrieval is close to or even outperforms some non-random retrieval methods.

Author response:
The observation regarding random retrieval occasionally outperforming strategized retrieval, particularly in Java, is indeed intriguing. We speculate that this phenomenon is influenced by the inherent characteristics of the language. In Java, cross-file references often involve class definitions (e.g., `public class DerivedClass extends BaseClass {`). In this scenario, previous lines might not provide contextual clues (meaningful information) for retrieval. While in Python, cross-file references may often involve more direct contextual clues such as function calls with parameters, making strategized retrieval more effective. For example, if you try to use `calculate(left, right)`, you may define `left=?` and `right=?` in the previous lines. Thus, our findings suggest that effective retrieval strategies may vary between different programming languages.

Reviewer comment (Writing):
What is exactly "the first appearance of a cross-file line within a file"? Is this the import line? Is this the first line that uses cross-file function?

Author response:
We acknowledge for the ambiguity in our description. The "first appearance of a cross-file line within a file" refers to the first instance where a cross-file function/class is used in the code, not the import line. So strictly speaking, it should be the second appearance (the first one will always be the import line).