Paper: BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual Pragmatic Knowledge (ICLR-2024)

Reviewer comment (Experiments):
Abstract. I would expect that in the second part of the abstract (after you motivated the problem of difficulty of correctly generating code for domain specific areas such as bioinformatics) you provide some evidence on why your dataset is either: (a) useful to help train models that achieve better performance for the given domain or (b) it provides a good measure to estimate whether an LLM is effective for the code generation in the field of bioinformatics (or specific subarea of bioinformatics coding domains).

Author response:
> Abstract. I would expect that in the second part of the abstract (after you motivated the problem of difficulty of correctly generating code for domain specific areas such as bioinformatics) you provide some evidence on why your dataset is either: (a) useful to help train models that achieve better performance for the given domain or (b) it provides a good measure to estimate whether an LLM is effective for the code generation in the field of bioinformatics (or specific subarea of bioinformatics coding domains).

> Abstract. Sentence "In relation to function-code generation ..." is unclear given the previous sentences. Reformulate.

> Abstract. Sentence "It incorporates 1026 functions and 1243 methods in Python and Java from GitHub". You probably need to add "respectively" after the word "Java".

Thank you for all your good suggestions on our abstract. We found it really useful, and we have edited our abstract based on them.

Current abstract:

>> Pre-trained large language models have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Bioinformatics provides an important domain. In this field generating functional programs poses additional notable challenges due to the amount of specialized domain knowledge, the need for complicated data operations, and intricate functional dependencies between the operations. Here, we present BioCoder, a benchmark developed to evaluate existing pre-trained models in generating bioinformatics code. In relation to function-code generation, BioCoder covers potential package dependencies, class declarations, and global variables. It incorporates 1026 functions and 1243 methods in Python and Java from GitHub and 253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing framework for evaluation, and we have applied it to evaluate many models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4. The results highlight two key aspects of successful models: 1) that they contain specific domain knowledge of bioinformatics (beyond just coding knowledge); 2) that they accommodate a long prompt with full context (i.e. functional dependencies).

Based on your comment, we could edit our abstract to:

>> Pre-trained large language models have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target the bioinformatics domain due to the amount of specialized domain knowledge, algorithms, and data operations. We present BioCoder, a benchmark developed to evaluate large language models (LLMs) in generating this bioinformatics-specific code. BioCoder spans a broad spectrum of the field and covers cross-file dependencies (e.g., package dependencies), class declarations, and global variables. It incorporates 1026 Python functions and 1243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling we show that overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate many models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4.  Furthermore, we finetuned StarCoder, demonstrating how our benchmark dataset can effectively enhance the performance of an LLM. The results highlight two key aspects of successful models: (1) that they contain specific domain knowledge of bioinformatics (beyond just coding knowledge); (2) that they accommodate a long prompt with full context (i.e. functional dependencies).

Reviewer comment (Writing):
Abstract. Sentence "In relation to function-code generation ..." is unclear given the previous sentences. Reformulate.

Author response:
> Abstract. I would expect that in the second part of the abstract (after you motivated the problem of difficulty of correctly generating code for domain specific areas such as bioinformatics) you provide some evidence on why your dataset is either: (a) useful to help train models that achieve better performance for the given domain or (b) it provides a good measure to estimate whether an LLM is effective for the code generation in the field of bioinformatics (or specific subarea of bioinformatics coding domains).

> Abstract. Sentence "In relation to function-code generation ..." is unclear given the previous sentences. Reformulate.

> Abstract. Sentence "It incorporates 1026 functions and 1243 methods in Python and Java from GitHub". You probably need to add "respectively" after the word "Java".

Thank you for all your good suggestions on our abstract. We found it really useful, and we have edited our abstract based on them.

Current abstract:

>> Pre-trained large language models have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Bioinformatics provides an important domain. In this field generating functional programs poses additional notable challenges due to the amount of specialized domain knowledge, the need for complicated data operations, and intricate functional dependencies between the operations. Here, we present BioCoder, a benchmark developed to evaluate existing pre-trained models in generating bioinformatics code. In relation to function-code generation, BioCoder covers potential package dependencies, class declarations, and global variables. It incorporates 1026 functions and 1243 methods in Python and Java from GitHub and 253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing framework for evaluation, and we have applied it to evaluate many models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4. The results highlight two key aspects of successful models: 1) that they contain specific domain knowledge of bioinformatics (beyond just coding knowledge); 2) that they accommodate a long prompt with full context (i.e. functional dependencies).

Based on your comment, we could edit our abstract to:

>> Pre-trained large language models have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target the bioinformatics domain due to the amount of specialized domain knowledge, algorithms, and data operations. We present BioCoder, a benchmark developed to evaluate large language models (LLMs) in generating this bioinformatics-specific code. BioCoder spans a broad spectrum of the field and covers cross-file dependencies (e.g., package dependencies), class declarations, and global variables. It incorporates 1026 Python functions and 1243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling we show that overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate many models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4.  Furthermore, we finetuned StarCoder, demonstrating how our benchmark dataset can effectively enhance the performance of an LLM. The results highlight two key aspects of successful models: (1) that they contain specific domain knowledge of bioinformatics (beyond just coding knowledge); (2) that they accommodate a long prompt with full context (i.e. functional dependencies).

Reviewer comment (Writing):
Abstract. Sentence "It incorporates 1026 functions and 1243 methods in Python and Java from GitHub". You probably need to add "respectively" after the word "Java".

Author response:
> Abstract. I would expect that in the second part of the abstract (after you motivated the problem of difficulty of correctly generating code for domain specific areas such as bioinformatics) you provide some evidence on why your dataset is either: (a) useful to help train models that achieve better performance for the given domain or (b) it provides a good measure to estimate whether an LLM is effective for the code generation in the field of bioinformatics (or specific subarea of bioinformatics coding domains).

> Abstract. Sentence "In relation to function-code generation ..." is unclear given the previous sentences. Reformulate.

> Abstract. Sentence "It incorporates 1026 functions and 1243 methods in Python and Java from GitHub". You probably need to add "respectively" after the word "Java".

Thank you for all your good suggestions on our abstract. We found it really useful, and we have edited our abstract based on them.

Current abstract:

>> Pre-trained large language models have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Bioinformatics provides an important domain. In this field generating functional programs poses additional notable challenges due to the amount of specialized domain knowledge, the need for complicated data operations, and intricate functional dependencies between the operations. Here, we present BioCoder, a benchmark developed to evaluate existing pre-trained models in generating bioinformatics code. In relation to function-code generation, BioCoder covers potential package dependencies, class declarations, and global variables. It incorporates 1026 functions and 1243 methods in Python and Java from GitHub and 253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing framework for evaluation, and we have applied it to evaluate many models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4. The results highlight two key aspects of successful models: 1) that they contain specific domain knowledge of bioinformatics (beyond just coding knowledge); 2) that they accommodate a long prompt with full context (i.e. functional dependencies).

Based on your comment, we could edit our abstract to:

>> Pre-trained large language models have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target the bioinformatics domain due to the amount of specialized domain knowledge, algorithms, and data operations. We present BioCoder, a benchmark developed to evaluate large language models (LLMs) in generating this bioinformatics-specific code. BioCoder spans a broad spectrum of the field and covers cross-file dependencies (e.g., package dependencies), class declarations, and global variables. It incorporates 1026 Python functions and 1243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling we show that overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate many models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4.  Furthermore, we finetuned StarCoder, demonstrating how our benchmark dataset can effectively enhance the performance of an LLM. The results highlight two key aspects of successful models: (1) that they contain specific domain knowledge of bioinformatics (beyond just coding knowledge); (2) that they accommodate a long prompt with full context (i.e. functional dependencies).

Reviewer comment (Experiments):
Introduction. I would like to see some discussion or evidence of why the dataset you constructed is an effective dataset for the field of bioinformatics. Some of the questions that are still left in my mind are:
- What is the range of problems/areas bioinformatics code typically encompasses? 
- What position/importance in the field of bioinformatics is occupied by the tasks you include in your dataset? I see some information in the appendix, but ultimately this is a crucial factor in the paper, so there needs to be a good evidence about it in the main paper.
- Ultimately, what is the value that a bioinformatics programmer, who wants to use an LLM to help writing his/her code, will gain by evaluating LLMs on your dataset? Can he/she gain some level of confidence that LLM that works well on your dataset will perform better on his/her problems?

Author response:
> Introduction. I would like to see some discussion or evidence of why the dataset you constructed is an effective dataset for the field of bioinformatics. Some of the questions that are still left in my mind are: What is the range of problems/areas bioinformatics code typically encompasses?

Our benchmark data encapsulates the majority of daily tasks a bioinformatician encounters in data analysis.  In Appendix O, Using topic modeling we show that overall coverage of the included code is representative of the full spectrum of bioinformatics calculations.

Bioinformatics code typically encompasses a wide array of problems targeted at understanding biological data. This includes genomic sequence analysis, predicting protein structure and function, analyzing gene expression, and protein-protein interactions, and developing algorithms and statistical models to interpret biological data, among others.


> What position/importance in the field of bioinformatics is occupied by the tasks you include in your dataset? I see some information in the appendix, but ultimately this is a crucial factor in the paper, so there needs to be a good evidence about it in the main paper.

The tasks covered in the \textsc{BioCoder} dataset were carefully extracted from popular bioinformatics packages of published research literature. We strived to extract a representative sample from these repositories included in survey papers, therefore ensuring that we cover as much of the bioinformatics field as possible. While we intentionally do not focus on a specific topic, we have ensured functions from high-profile repositories, such as those that deal with DNA sequencing, protein structure analysis, and gene expression analysis, are included. Therefore, we ensure a broad range of topics, guaranteeing that the most important tasks in the field of bioinformatics are covered. This ensures that our benchmark reflects real-world scenarios across a broad spectrum of bioinformatics tasks.

> Ultimately, what is the value that a bioinformatics programmer, who wants to use an LLM to help writing his/her code, will gain by evaluating LLMs on your dataset? Can he/she gain some level of confidence that LLM that works well on your dataset, will perform better on his/her problems?

BioCoder acts as a gauge of the potential performance of LLMs in the bioinformatics domain. If an LLM performs well on this dataset, it indicates that it has a good grasp of bioinformatics programming tasks and, thus, is likely to perform well on similar real-world tasks. This is compounded by the fact that our dataset places a special emphasis on cross-file dependencies, reflecting the development experience in a real-world scenario. This can help bioinformatics programmers decide which models to use, and how to use them, saving time and resources.

Reviewer comment (Experiments):
Introduction. You mention that you create "a new high-quality dataset". It may be a high-quality dataset, but it would be good to add some specific numbers or evidence on why it is so.

Author response:
> Introduction. You mention that you create "a new high-quality dataset". It may be a high-quality dataset, but it would be good to add some specific numbers or evidence on why it is so.

Thank you for your suggestion. While it is difficult to comment on the “quality” of the data with numbers and metrics, it is important to note the results in the preliminary human study, in which human annotators rated nearly all of the functions as high quality, and completable by a human. We promise to add finished results to the camera-ready version of the paper.
For now, we have updated the introduction in the manuscript accordingly.

ORIGINAL:

>> We create a new high-quality dataset for code generation, curated from 1,720 bioinformatics repositories referenced in peer-reviewed bioinformatics articles, aiding in the practical and realistic evaluation of code generation tasks.

NEW:

>> We create a new high-quality dataset for code generation, curated from 1,720 bioinformatics repositories referenced in peer-reviewed bioinformatics articles, aiding in the practical and realistic evaluation of code generation tasks. 
To ensure we captured the highest quality functions from these repositories, we employed two rounds of filtering,  with both an automatic approach (using GPT) and a manual examination. In the first round, we used GPT to compare each candidate function to a list of bioinformatics keywords, and then of the functions that matched the closest to the list, we performed a manual inspection to ensure it meets our guidelines of being highly bioinformatics-related, the non-trivial function that relies on code context. For the second round, it was done with the expertise of four students specializing in computer science and computational biology.

More information can be found in Section 3.1: DATASET FILTERING.

Reviewer comment (Writing):
"golden code file". While one can guess what the meaning of the golden code file is, it would be good to explain. This phrase is used only at a single place in the paper.

Author response:
> "golden code file". While one can guess what the meaning of the golden code file is, it would be good to explain. This phrase is used only at a single place in the paper.

Thank you so much for the feedback. Sorry for the confusion, we should use “golden code” instead of “‘golden code file’”. Here “golden code” is the ground truth code that is guaranteed to be correct, producing the correct outputs when run with the test cases. We use the golden code outputs to compare with the generated outputs for Pass @K testing purposes. Golden code and golden code files are used interchangeably to reference the original code from the repository, as opposed to the generated code from LLMs. We have updated the revision to reflect this.

Reviewer comment (Writing):
"Summary Only prompts". When encountering this phrase in the paper for the first time, it's not obvious what this means. One can look at Figure 3 and infer, but it would be good to have a natural flow in the main body of the paper itself.

Author response:
> "Summary Only prompts". When encountering this phrase in the paper for the first time, it's not obvious what this means. One can look at Figure 3 and infer, but it would be good to have a natural flow in the main body of the paper itself.
Thanks for your suggestion and we have updated Section 5. 
Summary-only prompts are shortened prompts utilized across all our LLM models to ensure that context-limited LLMs still receive all the information necessary to potentially generate functions. Within the summary-only prompts, we optimized our prompts to only contain the absolute minimum of necessary information, without including much of the additional context that provides detail regarding the functionality of other dependencies.

Reviewer comment (Presentation):
"Summary Only prompts". If your concern is space, I would rather remove results of certain prompt types from the main paper, and put them into the appendix. Then, for a single version of the prompt I would explain in detail in the main paper how it is constructed.

Author response:
> If your concern is space, I would rather remove results of certain prompt types from the main paper, and put them into the appendix. Then, for a single version of the prompt I would explain in detail in the main paper how it is constructed.

Thanks for your suggestions. We tried moving a subset of the results of prompt types to the Appendix, however since we repeatedly cite the results across various prompt types in the results analysis section, we could not find a way to make it coherent. For now, we keep the results of all prompt types for this revision of the paper, but we will look into this further. 

Furthermore, we have put a brief description of the prompt construction in Section 4. MODELS AND RESULTS.

Reviewer comment (Presentation):
"Summary Only prompts". Also, Figure 3 is not elaborated inside of the paper.

Author response:
> Also, Figure 3 is not elaborated inside of the paper.

Thanks for pointing it out. We edited the paper with new sections, and have now referenced Figure 3 in Section 3.2 and the second paragraph of Section 4. MODELS AND RESULTS.

Reviewer comment (Writing):
Table 4. "Results are in %". I would write something like: results are expressed in percentage.

Author response:
> Table 4. "Results are in %". I would write something like: results are expressed in percentage.

Thanks for pointing it out. We have updated it in the main text.

Reviewer comment (Presentation):
I would add line numbers for the review purposes so reviewers can easily refer to a given part of the paper in their comments.

Author response:
> I would add line numbers for the review purposes so reviewers can easily refer to a given part of the paper in their comments.

Thanks for suggesting it. We are using the ICLR official template, without line numbers.

Reviewer comment (Presentation):
Table 1. Spell out the abbreviations somewhere for P.C., P.L., C.C., C.L.

Author response:
> Table 1. Spell out the abbreviations somewhere for P.C., P.L., C.C., C.L.

Thank you so much for the feedback. We have updated the manuscript so that it more clearly indicates the meaning of the “summary only” prompts, and have optimized it so that it is much easier to follow.

Reviewer comment (Presentation):
Table 6. I would suggest using percentage instead of raw numbers for showing distribution of test error types. You can also include information about total number of test cases vs number of failed cases. You could possibly restructure the table to remove repetitive "Failure Reason".

Author response:
> Table 6. I would suggest using percentage instead of raw numbers for showing distribution of test error types. You can also include information about total number of te st cases vs number of failed cases. You could possibly restructure the table to remove repetitive "Failure Reason".

Thank you for your suggestion. We have updated table 6 to show the distribution of failure reasons in percentages, and merged some of the tables accordingly. They are reflected in the revision of the paper.

| Type                  | Quantity | Percentage |
|-----------------------|----------|------------|
| Mismatched output     | 8661     | 4.567%     |
| Invalid syntax        | 117665   | 62.038%    |
| Runtime error         | 55351    | 29.184%    |
| Time out              | 4        | 0.002%     |
| Successfully Passed   | 7982     | 4.209%     |



Once again, we **appreciate** your feedback and believe that our revisions have addressed your concerns. Your insights have helped improve the quality and clarity of our work.

Reviewer comment (Presentation):
The article includes an appendix that might be too long for the readers, and the content in the appendix is referred from the main body multiple times. It is better to make the paper more self-contained if it is accepted to be published in conference proceedings.

Author response:
Thank you for your constructive feedback. We agree that the paper could benefit from being more self-contained. As per your suggestion, we have revisited the main text and revised it in a way so as to include more key details that were initially placed in the appendix, for instance, the definition of different versions of Prompts. We are confident, after these revisions, that the paper stands comprehensive on its own. Even in the absence of the appendix, the purpose, methodologies, and key findings of our research can be entirely understood from the main text. We appreciate your insights and believe these changes significantly enhance the paper's readability and comprehensibility.

Reviewer comment (Novelty):
The author should clearly point out the main technical contribution of this paper.

Author response:
Thank you for both questions. We will answer them at the same time.

The technical contributions of this paper are multifold, primarily focusing on the development of tools, datasets, models, and testing environments to enhance bioinformatics code generation:

Code Parsing Tools – We've developed comprehensive tools for parsing large-scale code repositories, which include AST (Abstract Syntax Tree) parsers and utilities to extract code attributes. These tools facilitate efficient and systematic extraction, transformation, and storage of coding tasks from raw source codes. Furthermore, they also feature techniques to correct minor errors that arise during generation.

Data Processing Scripts - We have developed methodical procedures and scripts for data processing, which are both robust and transparent. These are provided so that others within the community may utilize and adapt them for their own tasks, lending methodological insights and fostering further research within the field.

Testing and Docker – We've established a robust testing environment that includes Docker configurations, required dependencies, and an array of fuzz test cases. The aim is to create a reproducible testing setup that not only imitates realistic project scenarios but also ensures the scalability and transferability of our approach. 

Model Zoo - In our effort to promote applicability and ease-of-use, we have created a 'Model Zoo.' This feature provides immediate access to numerous pre-implemented code-generation large language models, such as SantaCoder, StarCoder, and CodeGen alongside APIs based on OpenAI. This enables other researchers or developers to effortlessly test their tasks on many different models, without the overhead of implementing these models from scratch. We firmly believe that our model hub significantly contributes engineering-wise to the entire domain.

Reviewer comment (Writing):
The reviewer does not quite catch the challenges for benchmarking bioinformatics code generation compared to other domain specific languages.

Author response:
The challenges of benchmarking bioinformatics code generation originate from both the characteristics of the domain and the nature of the tasks involved. 

Good bioinformatics coding requires a deep understanding of both biological concepts and advanced computation methods. Generating or evaluating code thus requires knowledge and experience in both. Bioinformatics involves numerous subfields and types of tasks, from sequence alignment, and phylogenetic analysis, to protein structure prediction, to name just a few.

Hence, to generate high-quality and functional bioinformatics code, the model must be well-versed not only in programming but also in biological and computational concepts relevant to bioinformatics.

Reviewer comment (Evaluation):
How does the benchmarking for bioinformatics code generation differ from the benchmarking for other domain specific languages?

Author response:
While the concept of Bioinformatics code does not differ significantly from domains such as chemistry, it differs a bit from more mathematical and computational heavy domains like engineering, physics, and others. In our findings, the code is a lot more dependent on knowledge within the domain, and it is nearly impossible to generate any of the functions without the required domain knowledge.

Reviewer comment (Novelty):
What is the key technical contribution for the BioCoder that is strongly related to benchmarking bioinformatics code generation?

Author response:
Thank you for both questions. We will answer them at the same time.

The technical contributions of this paper are multifold, primarily focusing on the development of tools, datasets, models, and testing environments to enhance bioinformatics code generation:

Code Parsing Tools – We've developed comprehensive tools for parsing large-scale code repositories, which include AST (Abstract Syntax Tree) parsers and utilities to extract code attributes. These tools facilitate efficient and systematic extraction, transformation, and storage of coding tasks from raw source codes. Furthermore, they also feature techniques to correct minor errors that arise during generation.

Data Processing Scripts - We have developed methodical procedures and scripts for data processing, which are both robust and transparent. These are provided so that others within the community may utilize and adapt them for their own tasks, lending methodological insights and fostering further research within the field.

Testing and Docker – We've established a robust testing environment that includes Docker configurations, required dependencies, and an array of fuzz test cases. The aim is to create a reproducible testing setup that not only imitates realistic project scenarios but also ensures the scalability and transferability of our approach. 

Model Zoo - In our effort to promote applicability and ease-of-use, we have created a 'Model Zoo.' This feature provides immediate access to numerous pre-implemented code-generation large language models, such as SantaCoder, StarCoder, and CodeGen alongside APIs based on OpenAI. This enables other researchers or developers to effortlessly test their tasks on many different models, without the overhead of implementing these models from scratch. We firmly believe that our model hub significantly contributes engineering-wise to the entire domain.

Reviewer comment (Reproducibility):
My biggest concern is the lack of justifications for the testing framework for Python and Java functions in BioCoder. The authors state, "... we employ a custom syntax to indicate the insertion points for custom randomly generated test cases." How are the test cases "randomly generated" and inserted into the context files? I did not find any detailed explanations in the main paper or in Appendix L and Y. In particular, Appendix L only briefly introduces the pipeline of the testing framework, how does such an approach deliver "a secure and efficient testing framework, promising robustness in the evaluation of generated code"? The authors need to clarify more about the generation of the test cases.

Author response:
Thank you for your comments. We have addressed each one as follows:

> My biggest concern is the lack of justifications for the testing framework for Python and Java functions in BioCoder. The authors state, "... we employ a custom syntax to indicate the insertion points for custom randomly generated test cases." How are the test cases "randomly generated" and inserted into the context files? I did not find any detailed explanations in the main paper or in Appendix L and Y. In particular, Appendix L only briefly introduces the pipeline of the testing framework, how does such an approach deliver "a secure and efficient testing framework, promising robustness in the evaluation of generated code"? The authors need to clarify more about the generation of the test cases.

Apologies for the oversight, as we did not provide enough information about the testing framework in Appendix L. We provided a brief overview of the pipeline in Appendix L, but we will make sure to explain the details and importance of the framework in more depth.

Our fuzz testing framework was created based on the need to accurately compare ground truth and generated code samples, while optimizing the test case writing pipeline. A detailed explanation of the entire process, as well as our motivations, is written in the postceding paragraph. Our test cases can be thought of as typical unit tests with randomly generated parameters, hence why we are “inserting” randomly generated test cases into the code (more details given later). Our entire process, including the use of docker containers, virtualization, and consistent testing environments, allows us to ensure the security and robustness of our testing framework.

Here is a more detailed explanation:

We decided to utilize concepts from fuzz testing, as fuzz testing is widely used in the industry to capture bugs, crashes, security vulnerabilities, etc. in functions. However, in these cases, they do not have access to a “correct” version of the function; instead, they are merely creating inputs to intentionally try to crash the program, find out-of-bounds memory accesses, etc. Our situation is unique because we have the “golden code”, or the ground truth version of the code, so given input, we definitely know what the expected output should be, which is not something that’s usually available in typical fuzz testing frameworks. Therefore, our situation could be considered a mixture of both unit testing and fuzz testing.

Given this requirement, and the goal of large-scale prompt generation, we decided to implement our own framework. We set out to accomplish two things: make the annotation process a lot easier for human editors, and support our feature set that combines both elements from unit testing and elements from fuzz testing. We believe that our resulting pipeline is more intuitive than piecing together other testing frameworks, and in our annotation process, it proved to make things efficient, enabling larger-scale annotation, as the goal of the paper.

Furthermore, note that while handwritten test cases would likely target edge cases of a program (eg. branch coverage, conditional coverage), the probability of our fuzz testing framework hitting all of the same edge cases is high given 1000 iterations of randomly generated inputs. This means that we can save a significant amount of time building the dataset, as we only need to write an outline of a test case, and let the framework handle the rest. Therefore, we can think of the framework as thousands of “random unit tests,” with a high probability that these unit tests would include handwritten test cases, if we had written them.

In terms of variable generation, we replace the <|var_type;parameter|> syntax with random values each iteration, for an unlimited number of iterations. These parameters are modifiable, and we implemented this system to be flexible, so that we can target specific scopes for fuzz testing. We check correctness by substituting the exact same variables in the original code, and checking if the outputs of the two functions match. This indicates identical functionality to the original code.

Reviewer comment (Evaluation):
In addition to the previous point, for Rosalind functions, the authors mentioned, "...the output of this execution is compared with the cached golden code output." Why and how are the generated codes compared with the gold code outputs? I do not find any experiment results that illustrate the comparison outcomes.

Author response:
> In addition to the previous point, for Rosalind functions, 
the authors mentioned, "...the output of this execution is compared with the cached golden code output." 
Why and how are the generated codes compared with the gold code outputs? I do not find any experiment results that illustrate the comparison outcomes.


Thank you for the clarifying question. In the current paper, we have: “For Rosalind functions, the process is simpler and more efficient as the functions are less complex. The golden code’s output is generated and cached ahead of time. During testing, the tester executes the generated code within the corresponding context, and the output of this execution is compared with the cached golden code output.”

We admit that this is not clear. The “cached golden code output” refers to an optimization specific to Rosalind. We utilize the fact that we are given the same input for all test cases, and only run the golden code once, instead of running it 20 times in all 20 pairs of generated-golden code pairs. For the next revision, we will likely remove this in the name of clarity, and because it is an implementation detail that does not affect any results.

Rosalind (https://rosalind.info/) is an educational resource and web project for learning bioinformatics through problem-solving and computer programming.

In Rosalind Online Judge, they declare a code sample as “correct” using just one test case. We define “golden code” as the original code, which is guaranteed to be correct. Therefore, for our evaluation, we also use one test case, and use it as input for both the original golden code and the generated code. Then, the (typically string) outputs are compared, and declared “correct” if and only if it is an exact match.

In the current paper, the “cached golden code output” refers to the optimization where we used the same test case on each pair of golden and generated code. We used a web scraper to automatically download the desired “test cases” from the Rosalind website. Therefore, the correct output should always be the same between all executions of a problem, so we only need to execute the golden code once, and “cache” the output.

Based on this we can revise the paper to better explain the Rosalind testing process, and the similarities to the other testing process:

“For Rosalind functions, the process is simpler and more efficient as the functions are less complex. During testing, the tester executes the generated code and golden code within the corresponding context with the same input, and the output of the pair of executions is considered correct if and only if they are identical.”

Golden Code:
count the number of times that “A” appears in the string

```
def count_A(input_str):
    return input_str.count(“A”)
    print(count)
```

Generated code:
count the number of times that “A” appears in the string

```
def count_A(input_str):
    count=0
    for character in input_str:
        if character==’A’: count+=1
    print(count)
```

In these simplified examples, we could pass in the string “ABCAABC@” into the function as input_str. Note that both functions would correctly print “3” as the response, so we can mark it as correct. Here is another example of a generated sample:

count the number of times that “A” appears in the string

```
def count_A(input_str):
    count=0
    for character in input_str:
        if “A” in input_str: count+=1
    print(count)
```

In this example, if we pass in the same string, then it would print “8”, which is clearly not the same as what the golden code printed, and therefore we would mark it as incorrect.

Reviewer comment (Evaluation):
Another concern is the implementation of correction mechanisms which rectify minor syntax and style errors. What kind of syntax and style errors can be considered "minor" with no impact on the functionality of the generated programs? As the authors take invalid syntax and runtime error as two major failure reasons in the following error distribution analysis, I recommend further justifying such correction mechanisms, which may affect the validity of the analysis results.

Author response:
> Another concern is the implementation of correction mechanisms which rectify minor syntax and style errors. What kind of syntax and style errors can be considered "minor" with no impact on the functionality of the generated programs? As the authors take invalid syntax and runtime error as two major failure reasons in the following error distribution analysis, I recommend further justifying such correction mechanisms, which may affect the validity of the analysis results.

Thank you for the questions and suggestions. We admit that we did not describe the implementation details of this correction mechanism clearly. 

We define “minor syntax errors” as errors that we are able to correct automatically with our heuristics. For example, StarCoder often “over-generates” code, so during one generation (even with dynamic generation limits), it could generate 3-4 functions at once, while we would only need the first one. In other words, the model doesn’t know when to stop generating.

In this case, we perform dynamic code analysis to potentially determine any relationships between functions, and retain the functions that are utilized in the target function, and remove all extraneous data. In other cases, we automatically correct mismatched brackets, missing semicolons, incorrect indentations, etc. so when inserted back into the context, it adapts accordingly.

The logic behind this is that in real-world scenarios, if we are able to deterministically correct model outputs, then we should do so. We believe this method is justified as it doesn’t make sense to reduce resulting model performance by something that can be adjusted without human intervention.

We do acknowledge that there could be more optimizations and heuristics we could implement to further enhance this error correction process, however, we did our best in generalizing the most common model output errors we could.

Reviewer comment (Experiments):
Table 4 summarizes the performance of the studied LLMs on BioCoder w.r.t 4 different types of prompt formats. However, the explanations of the different prompt versions are placed in Appendix I, which makes Table 4 hard to understand. Moreover, Appendix I only gives explanations with examples of the prompts in each version; nevertheless, I am looking for some high-level guidelines for the prompt design. Namely, how the five prompt versions are proposed? Are they from existing lectures or experimental experience? What are the characteristics of different prompt formats?

Author response:
> What are the guidelines while designing the 5 different prompt styles for the subject LLMs? (*)

Thank you for your question. We would be happy to go into some more detail as to how we designed the 5 different prompt styles described in the paper. 

When designing the prompts, we wanted to make sure the prompts had sufficient and necessary information to capture the essence of the function that we were trying to generate. Specifically, we wanted to design prompts that contained enough code context so that an experienced programmer would have the necessary tools to complete the function. 
Using these guidelines, we tried hundreds of different ideas for prompt structures, manually testing the performance of various structures using the OpenAI API, as well as locally run models.

Eventually, we ended up with 5 main types of prompts, each logically motivated by some observation we made during our experiments.

We begin with the “summary at bottom” prompts, which is the format used by most papers (we based it off of CoderEval, but it seems to be used in many other papers as well). From here, we also follow the instruction model, putting the summary at the top.
For the context, it’s important to note that models typically give higher “priority” to context later in the prompt. Therefore, the inter-file context needs to go before the intra-file context, which is more likely to be referenced by the target function. However, this context is not displayed in correct syntax, so it might have been important to make a distinction between the inter-file and intra-file context, so we commented out the inter-file ones.
During testing, we observed that this would cause the model to generate an excessive amount of comments, sometimes generating the entire function as a comment. We couldn’t figure out a way to consistently handle this during the dynamic analysis/error correction process, so we uncommented these, and most models seemed to behave more or less “normally”
The “Necessary context only” is human annotated context, which has been fully cleaned so that only the necessary information is included, so it is logical to test these. This is in contrast to the other prompt types, which are almost entirely automatically generated from a repository.
For the “summary only” prompts, we purely wanted to see whether the function would be completable without any context whatsoever.

We feel that these prompts represent a logical progression in our pursuit to extract the highest possible performance out of these models. While we do acknowledge that these may not be the perfect prompts for every model tested, our testing shows that they are some of the best out of the hundreds we have tried, and therefore should be sufficient to prompt LLMs for code generation.


>  However, the explanations of the different prompt versions are placed in Appendix I, which makes Table 4 hard to understand

We have introduced a paragraph in Section 4. MODELS AND RESULTS briefly discuss our prompt versions. We hope it provides an adequate amount of information to explain Table 4, while the more lengthy explanation is in Appendix I.

> Moreover, Appendix I only gives explanations with examples of the prompts in each version; nevertheless, I am looking for some high-level guidelines for the prompt design. Namely, how the five prompt versions are proposed? 

Thank you for your question. Our answer to your question “What are the guidelines while designing the 5 different prompt styles for the subject LLMs?” includes an answer for this.


> Are they from existing lectures or experimental experience?

> What are the characteristics of different prompt formats?

Thank you for both questions. We will answer them at the same time.

We created the prompt formats based on experimental experience. We set out to create prompt structures to test the specific performance characteristics of each model. We first decided on a set of characters to target, as shown in the following prompt types:
Uncommented/Summary at top/Summary at bottom - target performance on deciding which context to use, as there is intentionally some extraneous context not required for function
Summary Only - gather metrics on the level of dependency on context
Necessary Only - isolate purely the logical reasoning abilities of models, as we assume that all context is utilized.
For each of the goals, we experimented with slight modifications of each prompt structure until we found one that achieved the highest performance, best representing the performance of that specific characteristic.

Reviewer comment (Evaluation):
The discussion of the experiment results seems shallow to me. In section 5, the authors consider there is an inverse relationship between the length of the input prompts and the performance of the generated codes. However, from Table 4 and Appendix I, the Necessary only prompts have relatively shorter prompts but lower passing rates compared to uncommented and Summary at Top/Bottom in most of the studied LLMs. The author may elaborate more on the perspectives of prompt structures and contents instead of just the length of the prompts.

Author response:
> The discussion of the experiment results seems shallow to me. In section 5, the authors consider there is an inverse relationship between the length of the input prompts and the performance of the generated codes. However, from Table 4 and Appendix I, the Necessary only prompts have relatively shorter prompts but lower passing rates compared to uncommented and Summary at Top/Bottom in most of the studied LLMs. The author may elaborate more on the perspectives of prompt structures and contents instead of just the length of the prompts.

Thank you for your suggestion. In response, we have added a new appendix to the manuscript discussing more analysis of the performance with the perspective of prompt structures and contents. Please refer to the updated manuscript for details on the content of the appendix. We have also updated the “Analysis and Discussion” section in the manuscript. Initially, we wrote:

>> As shown by the scatterplots in Appendix J, it seems that on models with an average Pass@K score of at least 2%, there is an inverse relationship between the number of tokens in the prompt and the Pass@K score. Furthermore, for models such as SantaCoder and GPT, the performance fell sharply after around 500 tokens. This could be due to the massive amount of context “confusing” the models. When we look at the Necessary Only prompts, a similar trend occurs for longer contexts, but the phenomenon is not as pronounced, as on average these prompts are significantly shorter. Further discussion on the results of each model can be found in Appendix I.

We have now updated it to read as follows in the manuscript:

>> As shown by the scatterplots in Appendix k, on models with an average Pass@K score of at least 2%, there is an inverse relationship between the number of tokens in the prompt and the Pass@K score. Furthermore, for models such as SantaCoder and GPT models, the performance fell sharply after around 500 tokens. Despite this, model performance can not only be attributed to prompt length. 
We can see that even though the "Necessary Only prompts are relatively shorter when compared to the "Summary at Top" or "Uncommented" prompts, the Pass@k performance of the "Uncommented" prompts is in fact worse for many of the models. For more analysis regarding this and prompt structure in general, please refer to Appendix aa.

Reviewer comment (Presentation):
The "Summary At Bottom" results illustrated in Appendix U seem incomplete (no row for GPT-4).

Author response:
> The "Summary At Bottom" results illustrated in Appendix U seem incomplete (no row for GPT-4).

We have updated the results of GPT-4 in Appendix U.

Reviewer comment (Presentation):
From section 3.4, "Our testing framework starts with a manual review of selected functions, leading to the creation of a context file and a golden code file for each problem (see Figure 3)". I do not find how Figure 3 is correlated with the testing framework, Figure 17 in Appendix R may be a better example.

Author response:
> “From section 3.4, "Our testing framework starts with a manual review of selected functions, leading to the creation of a context file and a golden code file for each problem (see Figure 3)". I do not find how Figure 3 is correlated with the testing framework, Figure 17 in Appendix R may be a better example.”

Thank you for your feedback. We have corrected the reference in the revised paper.

Reviewer comment (Evaluation):
For Rosalind functions, why and how are the generated codes compared with the gold code outputs?

Author response:
> In addition to the previous point, for Rosalind functions, 
the authors mentioned, "...the output of this execution is compared with the cached golden code output." 
Why and how are the generated codes compared with the gold code outputs? I do not find any experiment results that illustrate the comparison outcomes.


Thank you for the clarifying question. In the current paper, we have: “For Rosalind functions, the process is simpler and more efficient as the functions are less complex. The golden code’s output is generated and cached ahead of time. During testing, the tester executes the generated code within the corresponding context, and the output of this execution is compared with the cached golden code output.”

We admit that this is not clear. The “cached golden code output” refers to an optimization specific to Rosalind. We utilize the fact that we are given the same input for all test cases, and only run the golden code once, instead of running it 20 times in all 20 pairs of generated-golden code pairs. For the next revision, we will likely remove this in the name of clarity, and because it is an implementation detail that does not affect any results.

Rosalind (https://rosalind.info/) is an educational resource and web project for learning bioinformatics through problem-solving and computer programming.

In Rosalind Online Judge, they declare a code sample as “correct” using just one test case. We define “golden code” as the original code, which is guaranteed to be correct. Therefore, for our evaluation, we also use one test case, and use it as input for both the original golden code and the generated code. Then, the (typically string) outputs are compared, and declared “correct” if and only if it is an exact match.

In the current paper, the “cached golden code output” refers to the optimization where we used the same test case on each pair of golden and generated code. We used a web scraper to automatically download the desired “test cases” from the Rosalind website. Therefore, the correct output should always be the same between all executions of a problem, so we only need to execute the golden code once, and “cache” the output.

Based on this we can revise the paper to better explain the Rosalind testing process, and the similarities to the other testing process:

“For Rosalind functions, the process is simpler and more efficient as the functions are less complex. During testing, the tester executes the generated code and golden code within the corresponding context with the same input, and the output of the pair of executions is considered correct if and only if they are identical.”

Golden Code:
count the number of times that “A” appears in the string

```
def count_A(input_str):
    return input_str.count(“A”)
    print(count)
```

Generated code:
count the number of times that “A” appears in the string

```
def count_A(input_str):
    count=0
    for character in input_str:
        if character==’A’: count+=1
    print(count)
```

In these simplified examples, we could pass in the string “ABCAABC@” into the function as input_str. Note that both functions would correctly print “3” as the response, so we can mark it as correct. Here is another example of a generated sample:

count the number of times that “A” appears in the string

```
def count_A(input_str):
    count=0
    for character in input_str:
        if “A” in input_str: count+=1
    print(count)
```

In this example, if we pass in the same string, then it would print “8”, which is clearly not the same as what the golden code printed, and therefore we would mark it as incorrect.

Reviewer comment (Presentation):
Explanations of the prompt configurations, shown in Table 4, should come in the Table description or somewhere in the main text, not only in the Appendix.

Author response:
We have modified the paper accordingly to better organize a brief description of the prompt configurations into the main sections of the paper, specifically, in the second paragraph of Section 4. MODELS AND RESULTS.

Reviewer comment (Experiments):
It would be interesting to have a human evaluation or experiment considering the descriptions as a way to bring more validation to the GPT3.5 creation.

Author response:
Furthermore, we have conducted a preliminary human evaluation study on the quality of the GPT-generated summaries. We have randomly sampled 50 summaries that we generated and have had 5 undergraduate students majoring in Computer Science classify each summary, and determine whether there are any errors in their description of the function. 

In this preliminary study, we have found that 48/50 summaries are specific enough to complete a function without much ambiguity in functionality. All summaries accurately reflect the intention of the function, tested by asking whether they could write the code themselves, after reading the summary. This demonstrates the accuracy of the summaries. We do acknowledge that our sample size of 50 summaries is somewhat small, but we wanted to provide a timely response, and we believe that our sample is representative of the larger dataset of summaries.