Paper: xCodeEval: An Execution based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (ICLR-2024)

Reviewer comment (Novelty):
The contribution of the paper is spread a bit thin. On the dataset side, while the dataset will likely be useful for program synthesis, given the existence of other similar datasets (although they either have fewer languages or smaller scale, see S1), more work is needed to verify that the scale and multi-lingual nature of the dataset is a useful feature.

Author response:
**W1)** Please see our General response on Novelty. In addition to that, we provide 62,798 unit tests (Table 2), more than double the number in previous benchmarks. These tests comprehensively cover all necessary scenarios to verify the correctness of a solution. The correctness, defined by a set of unit tests, is initially confirmed by a human annotator/problem setter when the problem is created on Codeforces and later validated by contestants and the problem-solving community. Thus the problems and unit tests are of high quality. We have also  ensured that our evaluation data includes the full unit tests, a step not taken in the CodeContest and APPS benchmarks. We believe this addresses a major shortfall in previous benchmarks and is rectified in ours. XCodeEval can be a good resource to train models for automatic test case generation, which is not within the scope of this (already dense) benchmark paper.

Reviewer comment (Evaluation):
For scale, it would be helpful to verify that the problems and tests are high quality and to consider automatically generating more tests, given that work like CodeContests [Li et al.] finds that many naturally-occurring tests have spotty coverage, leading to false positives.

Author response:
**W1)** Please see our General response on Novelty. In addition to that, we provide 62,798 unit tests (Table 2), more than double the number in previous benchmarks. These tests comprehensively cover all necessary scenarios to verify the correctness of a solution. The correctness, defined by a set of unit tests, is initially confirmed by a human annotator/problem setter when the problem is created on Codeforces and later validated by contestants and the problem-solving community. Thus the problems and unit tests are of high quality. We have also  ensured that our evaluation data includes the full unit tests, a step not taken in the CodeContest and APPS benchmarks. We believe this addresses a major shortfall in previous benchmarks and is rectified in ours. XCodeEval can be a good resource to train models for automatic test case generation, which is not within the scope of this (already dense) benchmark paper.

Reviewer comment (Experiments):
The experiments consisted mainly of ChatGPT with some additional experiments on open-source models. While the use of open-source models is appreciated, they were relatively small: CodeLlama-Instruct (up to 13B) and StarCoder (3B, fine-tuned). Given the very low performance of these models, and that a different model family was fine-tuned than used zero-shot, it is unclear what can be concluded from the finding that StarCoder-3b fine-tuned outperformed CodeLlama-7b Instruct.

Author response:
**W2a)** As of now, one of the main issues with the new genre of Large Language Models (LLMs) is that not all models are open-source. The way right now open-source and closed-source models “communicate” is via evaluation benchmarks. While open-source models are a significant part of the ecosystem, our proposed benchmark aims to ensure its applicability to any black-box (closed-source) model. The objective of the experiments presented in Table 5 is to demonstrate the impact of the finetuning data given along with the benchmark. The experiment in Table 5 reveals that a fine-tuned version of the 3B model can outperform a general-purpose 7B SFT model, but it cannot surpass a 13B general-purpose SFT model. Also this experiment highlights the trade-offs between model size and fine-tuning on specific corpora.

Reviewer comment (Evaluation):
The temperature analysis and "reasoning spectrum" did not appear to add much to the paper, as it was difficult to find a clear takeaway from them.

Author response:
**W2b)** Temperature is a critical hyperparameter in the decoding strategy of language models. Analysis involving temperature demonstrates a significant performance gap when decoding at different temperature values. Given that generating programs necessitates maintaining consistent reasoning in long-range sequences, we illustrate that model developers should consider this parameter attentively during evaluation.

The reasoning spectrum provides a singular visualization across the entire evaluation benchmark, comparing unit tests across languages. For instance, a vertical green line on the y-axis indicates that all languages can pass that particular unit test. Additionally, the reasoning spectrum offers a more detailed comparison of unit tests between languages.

Reviewer comment (Experiments):
The other tasks beyond program synthesis were under-explored. Code translation and APR, in particular, seem potentially useful (especially that APR is based on chronological submissions by a participant, as explained in E.5), but both had very limited experimental results. The difficulty of the retrieval task also seems to depend heavily on the size of the retrieval dataset (as evidenced in the text description of Table 4, with high vs. low resourced languages). Future work may need to introduce other metrics or account for dataset size.

Author response:
**W2c)**  Please refer to Table 3 for the complete results of the APR and Code-Translation Task. Due to spacing issues, a detailed description of all the tasks is also provided in Appendix E.

Reviewer comment (Experiments):
There is curiosity about the distribution of tags in the dataset, as this would give significant information about the fine-grained types of algorithms involved.

Author response:
**W2d)** The distribution of tags can be found in Appendix E.2. Additionally, the effect of our data selection strategy, as explained in Section 2, is detailed in Table 7.

Reviewer comment (Evaluation):
It is unclear how the top-k retrieval accuracy metrics account for there being multiple solutions in a given language per problem (if understood correctly).

Author response:
**Q2)** In our context, Top-k accuracy determines whether a PASSED solution exists within the top-k retrieval of the code for a query problem ID. We adopt the standard interpretation of retrieval accuracy in NLP and IR literature which denote the fraction of queries for which the top-k retrievals contain any of the corresponding solution/answer code. We refer to [BEIR](https://openreview.net/forum?id=wCu6T5xFjeJ) and [DPR](https://arxiv.org/abs/2004.04906) for more details.

Reviewer comment (Writing):
In the abstract, specify the source of the data (code contests problems).

Author response:
**Other Clarification**:
- **Source of the data**: The data was sourced from codeforces.com. We politely crawled the data over a period of six months. We did not use any data from code-contest or other sources.

Reviewer comment (Writing):
The description of "reasoning path" on page 8 was confusing, since it only evaluates the output of the code (PASSED vs. WRONG ANSWER, etc.), but "reasoning" conveys different algorithms or code implementations that all can produce PASSED solutions.

Author response:
- The description of "reasoning path" on page 8 was confusing. The program synthesis problem is evaluated through a set of unit tests. This collection of unit tests ensures coverage of all conditions required by the algorithms, including corner cases. Assume an algorithm necessitates five concepts of reasoning. A problem setter might implement these five concepts through 50 distinct unit tests. Therefore, a single unit test or a collection of them can define one or several reasoning paths.

Reviewer comment (Presentation):
While the paper appears to be comprehensive, it is extremely dense and not self-contained. It appears that the authors aimed at covering a ton of work and have to skip most details, resulting in most sections being cursory. Figures and tables are not presented clearly and barely readable. What's worse, the related work section is completely missing in the main body of the paper, which is unacceptable. The authors should consider how to streamline the content and make sure that the main paper is self-contained.

Author response:
**On the Clarity of the paper (Dense contents)**: In order to demonstrate the remarkable aspects and distinctive features of xCodeEval and our contribution, it is essential to depict the bottlenecks of the existing code benchmarks, how xCodeEval addresses them, our novel data selection schema based on a “Circulation problem with lower and upper bound strategy”, developing a full NL-Code benchmarking peipeline including our multilingual code execution engine, ExecEval, presenating data properties and uniqueness, all featured tasks, a comprehensive analysis and ablation study on the reasoning capabilities of LLM models considering the possible data leakage which is really a lot of contents to present.  Unfortunately, ICLR has a page limit. In Revision, we will improve the clarity of the paper and leverage the addition page. However we believe that in the current context, our paper is self-contained in terms of benchmark attributes, evaluation and analysis.

Reviewer comment (Experiments):
The dataset comes with finetuning data, however, it wasn't explained well why we need finetuning data at the era of LLMs. How would it add value in benchmarking an LLM?

Author response:
**Fine-Tuning Dataset**: We disagree with the claim that with LLs, the finetuning is no longer needed, rather finetuning models still prevail especially for adaptation to specific code-tasks or domains, enhancing performance, protecting data privacy and allowing benchmarking on code-specific tasks. It enables customization, addresses model analysis and so on. Nonetheless, the fine tuning dataset is specially intended to be used for finetuning smaller models considering that the evaluation benchmark may not be suited for zero-shot evaluation of such models (at least in today’s standard). In Figure 5, we show that fine-tuning a 3B model with training data can outperform a 7B general purpose SFT model. This shows that the training data can be used for enabling document level executability to smaller models. We also take extra precaution for train-dev-test overlap (Section 2.1) so that the integrity of the benchmark remains robust and training/finetuing data doesn’t interfere with benchmark evaluation. Also open sourcing fine tuning data helps the model evaluators to do data contamination analysis.

Reviewer comment (Reproducibility):
The license of the dataset is CC-BY-NC, which significantly limits the usability of the dataset, especially given most LLMs come from the industry. Is this coming from the authors or codeforces? If the former, it would be great if the authors can re-consider the decision.

Author response:
**License Issue**:

We want to emphasize that codeforces data already exists in popular benchmarks; CodeContest [1], APPS [2] and Avatar [3] use codeforces data for training and evaluation. We strongly believe that the license should not be a weakness factor for the paper as it does not restrict any research usage (as per the ICLR policy).

Reviewer comment (Experiments):
How does xCodeEval ensure the quality of its benchmarks across different languages?

Author response:
**How does xCodeEval ensure the quality of its benchmarks across different languages?**: We take the following measures to make sure the quality of the benchmark.
  - One of the main attributes of the generative evaluation of xCodeEval is that it is language agnostic. The output is evaluated by the unit tests. We make sure that our evaluation framework ExecEval can properly evaluate the languages that we advertise. We also add safety features using “prlimit” and “seccomp” at the time of code execution. 
  - All the problems in the codeforces do not have a complete set of test cases. Thus the previous benchmarks CodeContest [1] and APPS [2]  have incomplete unit test benchmarks. We make sure that all of our validation/test problems have the complete set of unit tests. More details are in the General Reply and “Evaluation and its granularity” section (Page 3) of the paper.
  - We make sure that validation-test distribution has a balanced distribution over the domain of the problems. We proposed a novel approach to create a balanced validation-test distribution using circulation problems with upper and lower bound in section 2. 
  - We have cleaned the corpus with automated tools  identifying and removing codes with sensitive details (i.e., email address), resulting in the removal of approximately 2 million samples from our original collection. We also followed all the necessary steps mentioned by Gebru et al [8], Holland et al [9], Hutchinson et al [10].

Reviewer comment (Experiments):
How would the new split algorithm (sec 2.1) improve over the baselines (e.g., random selection and/or time-based selection)?

Author response:
**Performance of the new split algorithm**: Section D2 with Table 7 in the Appendix shows that the new split algorithm (Sec 2.1) improves from the random baseline. We measured the skew, and standard deviation of the distribution of tags and showed that our proposed algorithm performs better than random data selection.

Reviewer comment (Experiments):
Did the authors obtain hidden tests (if any) from codeforces?

Author response:
**Unit test details**: Yes, all the hidden test sets are obtained from codeforces which are openly available. For validation and test datasets, we ignore the problems for which the test cases are incomplete. A complete collection of test sets ensures the correctness of a proposed solution..

Reviewer comment (Novelty):
The benchmark draws exclusively from Codeforces, which may limit the novelty of the dataset, as similar approaches have been used in other benchmarks like APPS.

Author response:
There are many fundamental issues we solve that exist in prior evaluation frameworks like CodeContest and APPS. Here are some description (also include in the general response and paper)
  - **How xCodeEval differs from APPS, CodeContests**: For CodeContests ([repository](https://huggingface.co/datasets/deepmind/code_contests/viewer/default/test?row=1)), 44 out of 165 test samples had no private unit tests. Additionally, there were inconclusive test cases. For instance, consider [this](https://codeforces.com/contest/1575/submission/130567445) problem in row 1 which contains 7 private test cases, while in the original codeforces competition, there are 34 test cases (accessible through the link here, you might need to log in and then click on Click to see test details), and some of them were incomplete (ended by ...) even in the codeforces API. We excluded problems with incomplete test cases from our development and test datasets. It's important to note that a problem is considered solved only when all the unit test cases pass successfully. We are grateful to our insightful reviewer for bringing up this aspect, and we plan to provide more context on these statistics in the final version of our paper. We mention that in “Evaluation and its granularity” section (Page 3). 
Similar problem occurs with APPS. For example: https://huggingface.co/datasets/codeparrot/apps (test split, row index 6) and codeforces source https://codeforces.com/contest/1217/problem/B, There are 365 test cases, while APPS included only 224. Note that these test cases, created by expert humans, as a whole ensure the correctness of a solution: a solution passed on all except one is considered as a wrong answer. xCodeEval ensures that this hypothesis preserves, while both APPS and CodeContest don’t ensure that. 
 - APPS and CodeContests offer only program synthesis tasks. xCodeEval proposes  5 different tasks from Classification, Generation and Retrieval tasks. From Table 1, APPS has  22,711 and  27,220 unit tests while xCodeEval covers 62,798 unit tests.
 - APPS and CodeContests problems do not provide some metadata, while xCodeEval provides the following crucial metadata:
   - **Timestamp**: Problem release date which helps to analyze data contamination (check Fig 4-left). This is indeed very crucial see this interesting [blog](https://lmsys.org/blog/2023-11-14-llm-decontaminator/)  and [paper](https://arxiv.org/pdf/2311.04850.pdf) on beating GPT-4! 
   - **Difficulty level**: Human annotated difficulty level in the range 800 to 3500).  APPS problems are divided into Competition, Interview and Introductory These are more like domains. For example, an Introductory problem can be harder than a Competition level problem. For example, in [APPS competition test, row 5](https://huggingface.co/datasets/codeparrot/apps/viewer/competition/test?p=9&row=904) has a difficulty of 1300 whereas in [APPS introductory test row 3](https://huggingface.co/datasets/codeparrot/apps/viewer/introductory/test) has a difficulty of 2100 at codeforces. Both of these diffuculty levels are annotated by human experts. 
   - **Tags/category**: xCodeEval comes from 31 different domains. Each of the domain problems can be identified during evaluation to understand about the domain biases of the code generation in LLM; see Figure 8 for details.

Reviewer comment (Experiments):
The reliance on a single platform like Codeforces means the benchmark might not capture the full spectrum of coding tasks. Basically the benchmark is limited to algorithm contest questions.

Author response:
In our General Response, we explain xCodeEval’s novelty and how it differs from other datasets. In addition, Table 6 shows that most of the datasets typically cover a single domain. Specifically, the popular module-level benchmarks like HumanEval, MBPP, MBXP, HumanEval-X, MultiPL-E and BabelCode evaluate language specific features which usually exist in book exercises. The DS-100, Exe-DS, ARCADE are mainly python-dependent execution based Data Science problems. Other than that, most of the competitive programming language based evaluation benchmarks [1][2][3][4][5] use problems from ICPC style problems. Though xCodeEval primarily uses codeforces problems, it has the largest ICPC style problem collection. In addition to the regular contest, codeforces regularly host contests for graduate university, past ICPC contests, educational rounds (traditional problems) as well as [legacy problems](https://codeforces.com/gyms). We want to emphasize that though codeforces sounds like a single site, it hosts the largest variety of problems which covers more than 31 domains of problems.

Reviewer comment (Evaluation):
The high difficulty level of the benchmark, as evidenced by low pass rates (less than 4%) across several 3B/7B/15B LLMs in Table 5, suggests it may be too challenging for all models with less than 30B parameters. This limits its utility for evaluating a broader range of model sizes and capabilities.

Author response:
Please see our general response. We strongly believe that the high difficulty level is indeed a strength instead of weakness. With the release of ChatGPT and GPT-4, it is evident that most of the benchmarks are either already memorized by the LLMs from the training data or the LLMs are generalized enough so that the benchmark becomes too easy. See this interesting blog  and paper why we need to rethink benchmarking and contamination for LLMs. 
We believe that a verifiable robust benchmark like xCodeEval can enable evaluation on the current genre of LLM as well as the next genre of more capable frontier models. (i) reasoning (ii) long-range document level multi-lingual seven code tasks (iii) following extreme sophisticated details and (iv) accounting for smallest errors that result in inaccuracy, and (v) a comprehensive analysis of model’s performances considering possible data leakage.

Reviewer comment (Presentation):
The visual presentation of the paper could be enhanced for better clarity and accessibility. Specifically, some graphics and tables, such as the legend in Figure 4, are difficult to read in print form.

Author response:
We will improve the visual presentation of the paper.

Reviewer comment (Experiments):
There are plenty of enhancements coming into code datasets with additional processing of GitHub-based datasets like Stack v2.

Author response:
We want to emphasize that the objective of **Stack v2** and **xCodeEval** dataset is completely different thus they cannot be compared with each other. Stack v2 is a pre-training (usually via self-supervised) corpus while xCodeEval is an (NL-Code, Code-Code annotated) evaluation benchmark taken from 25M samples using a novel data selection strategy. It comes with the largest set of languages (see Table 1 & Table 6 ) compared with other benchmarks. It also provides an additional training corpus which can be used to do fine tuning. The training corpus is specially required to enable executability to the smaller models.

Reviewer comment (Experiments):
The additional languages are also mostly mainstream and available in different datasets; it would have helped if the work included really low resource languages (Stack v2 has shown that too with even languages like COBOL).

Author response:
We want to emphasize that the objective of **Stack v2** and **xCodeEval** dataset is completely different thus they cannot be compared with each other. Stack v2 is a pre-training (usually via self-supervised) corpus while xCodeEval is an (NL-Code, Code-Code annotated) evaluation benchmark taken from 25M samples using a novel data selection strategy. It comes with the largest set of languages (see Table 1 & Table 6 ) compared with other benchmarks. It also provides an additional training corpus which can be used to do fine tuning. The training corpus is specially required to enable executability to the smaller models.

Reviewer comment (Experiments):
What would it take to expand the work to include other low resource languages including from a different domain?

Author response:
Since xCodeEval is a unit test based benchmark, the program synthesis and code translation tasks of xCodeEval can be extended to any number languages as long as the evaluation framework supports that language. The APR task can be modified to Cross-Lingual Program-repair task and extended to any number of languages. For retrieval and classification tasks, we need data for low resource languages to extend the evaluation framework. However, these tasks can also be used for contrastive representation learning.