Paper: Grounding Code Generation with Input-Output Specifications (ICLR-2024)

Reviewer comment (Novelty):
The idea is not novel enough.

Author response:
We appreciate your insight regarding the multi-task learning aspect of CodeT, where the LLM is prompted to generate an auxiliary task such as test cases to enhance the main task such as code generation. This perspective indeed draws connections between GIFT4Code and CodeT.

The conceptual difference is GIFT4Code addresses a different, yet equally important challenge: ensuring that LLMs follow the auxiliary tasks such as specifications or tests which are **already provided** in the prompt. This focus is fundamentally different from CodeT, which generates auxiliary tasks (tests) as part of its sampling process. In this context, GIFT4Code could potentially address an issue that might arise in CodeT, where the LLM fails to generate code satisfying its own previously generated test cases. Hence, we see GIFT4Code and CodeT as complementary approaches, each addressing different aspects of LLM-based code generation.

While we agree that the multi-task learning framework links CodeT and GIFT4Code, we believe that categorizing GIFT4Code as lacking novelty based solely on this broader perspective is not entirely fair. The multi-task learning formulation is a common theme in LLM research due to its emergent abilities, as represented by chain-of-thought which first generates reasoning as an auxiliary task then the main task. Our contribution with GIFT4Code is in how we apply this framework specifically to improve adherence to auxiliary tasks in code generation, which is a novel and important direction in the field. We hope this clarification further illustrates more details on the differences between CodeT and GIFT4Code.

Reviewer comment (Evaluation):
The authors claim that the code generated by these models occasionally deviates from the user’s intended outcome. However, CodeT also considers the test case, which is similar to the specification.

Author response:
Thank you for your review and the opportunity to clarify the aspects of our work.

> How is GIFT4Code fundamentally different from approaches like CodeT that also consider test cases similar to specifications? Can you elaborate on the differences between your approach and existing methods such as CodeT?

Thank you for pointing out that CodeT should be considered as a related work. CodeT focuses on using the generated tests to conduct dual execution agreements to filter the generated solutions in the inference time, when few candidates must be submitted. This is evident by the usage of pass@k@n metric, which selects the best k samples from a total n samples. Essentially, CodeT's goal is to refine the inference process by optimizing the selection of generated code samples. This approach is fundamentally different from GIFT4Code’s objective.

GIFT4Code aims to enhance the **base model’s** ability to follow complex I/O specifications. This difference in approach is reflected in our use of the pass@n metric. As a comparison, CodeT doesn’t improve pass@n as shown in Table 2 in the paper, “We do not show CODET pass@100, since it is the same as the baseline pass@100”. 

Moreover, in the realm of data science, the task of generating meaningful tests, as required by CodeT, can be particularly challenging. For instance, generating tests for a task like “What are the most populous cities in each country” (as shown in Figure 1) may not be feasible or meaningful. This complexity further distinguishes GIFT4Code's approach.

We will make sure to include a comparison with works that leverage execution or tests for filtering or clustering generated solutions during the inference time such as  [1, 2] in the revision. We hope this response addresses your concerns.

[1]: Chen, Bei et al. “CodeT: Code Generation with Generated Tests.” ArXiv abs/2207.10397 (2022): n. pag.

[2]: Zhang, Shun et al. “Planning with Large Language Models for Code Generation.” ICLR 2023.

Reviewer comment (Novelty):
Although there are known differences between the specification and the test case, the overall framework of these two approaches is similar, and if there is any difference, the authors should add more details about it.

Author response:
Thank you for your review and the opportunity to clarify the aspects of our work.

> How is GIFT4Code fundamentally different from approaches like CodeT that also consider test cases similar to specifications? Can you elaborate on the differences between your approach and existing methods such as CodeT?

Thank you for pointing out that CodeT should be considered as a related work. CodeT focuses on using the generated tests to conduct dual execution agreements to filter the generated solutions in the inference time, when few candidates must be submitted. This is evident by the usage of pass@k@n metric, which selects the best k samples from a total n samples. Essentially, CodeT's goal is to refine the inference process by optimizing the selection of generated code samples. This approach is fundamentally different from GIFT4Code’s objective.

GIFT4Code aims to enhance the **base model’s** ability to follow complex I/O specifications. This difference in approach is reflected in our use of the pass@n metric. As a comparison, CodeT doesn’t improve pass@n as shown in Table 2 in the paper, “We do not show CODET pass@100, since it is the same as the baseline pass@100”. 

Moreover, in the realm of data science, the task of generating meaningful tests, as required by CodeT, can be particularly challenging. For instance, generating tests for a task like “What are the most populous cities in each country” (as shown in Figure 1) may not be feasible or meaningful. This complexity further distinguishes GIFT4Code's approach.

We will make sure to include a comparison with works that leverage execution or tests for filtering or clustering generated solutions during the inference time such as  [1, 2] in the revision. We hope this response addresses your concerns.

[1]: Chen, Bei et al. “CodeT: Code Generation with Generated Tests.” ArXiv abs/2207.10397 (2022): n. pag.

[2]: Zhang, Shun et al. “Planning with Large Language Models for Code Generation.” ICLR 2023.

Reviewer comment (Experiments):
Unclear description regarding how to get I/O summary in the realistic setup. When the model is deployed for inference in the realistic setup, especially when only human intent is provided, it is not clear where the I/O summary could potentially come from, and it is no more the same case as training data preparation that there are some ground-truth programs to be executed for the output and summarized by the "generalist" LLM. Will Gift4Code generate the I/O summary first, by only conditioned on the problem description, and then generate the code conditioned on its own generated I/O summary? Or the user has to provide a concrete I/O specification beyond the high-level intent? In the latter case, whether the user could generate a high-quality I/O specification as the training data remains unknown. Though the authors explain that they try to simulate noisy I/O specification at test time (Section 4.1), this is still some sort of leakage of the ground-truth code, not similar to the realistic setup when only human intent is available. The authors are urged to explain how Gift4Code will be applied in such real-world scenarios.

Author response:
> Real-world setup of GIFT4Code?

After model deployment, we anticipate that developers will provide I/O specifications in their prompts, a similar assumption used in [1]. GIFT4Code is designed to tackle a prevalent challenge: even when complex I/O specifications are provided, models often fail to align with them, leading to inaccuracies in code generation. This is a significant issue in practice, where developers rely on the model to follow closely to the provided specifications for correct code generation. 

We conducted a preliminary experiment where the code LLM first generated an I/O summary and then the code solution. The pass@20 is ~49.3% which is lower than the results obtained with high quality but noisy specifications generated by the “generalist LLM” (55.5%). Acknowledging that real-world scenarios might not always offer explicit I/O summary, extending GIFT4Code to adapt to such conditions remains a key area for future work.

[1]: Shi, Kensen et al. “TF-Coder: Program Synthesis for Tensor Manipulations.” TOPLAS (2022)

Reviewer comment (Evaluation):
The evaluation of the paper is insufficient. While the design of Gift4Code seems effective, the evaluation is not sufficient.

Author response:
> The evaluation is insufficient. The baseline might be too weak.

Responding to your feedback on our evaluation process, we have conducted additional experiments to include two CodeLlama-34b models (LLama 2 70B is worse than CodeLlama 34b in coding tasks) in our evaluations. The following table presents the results from our extended evaluation on the full context Arcade benchmark:

| Model | Code LLM (no spec.) | CodeLlama-Python | CodeLlama-instruct |  Ours (synthetic f.t. w. I/O Summary) |
|-----|-----|-----|------|------|
| pass@20 	| 37.5 | 36.2 | 36.0 | 55.47 |

These results indicate that the base Code LLM, used as the starting point for GIFT4Code, is indeed a competitive model. Additionally, the comparable performance of the 15B StarCoder, 34B CodeLlama and the vanilla 62B Code LLM showed that the Arcade benchmark is very challenging, which could not be solved by just increasing the number of parameters, highlighting the unique value GIFT4Code brings to the data science domain.

Reviewer comment (Experiments):
The baselines are not strong enough. For the model of similar size, i.e., 62B, the authors only compare with an unknown/anonymous code LLM which is used as the starting point of Gift4Code. Given the audience could not estimate the capacity of this 62B since it is not publicly available, using this model as the only baseline seems insufficient. In addition, the 15B baselines seem to perform comparably well to the vanilla 62B Code LLM, making the latter, as a baseline, even weaker. The authors are encouraged to compare with at least one more publicly available model that is widely recognized as "capable", such as Llama-2-70B.

Author response:
> The evaluation is insufficient. The baseline might be too weak.

Responding to your feedback on our evaluation process, we have conducted additional experiments to include two CodeLlama-34b models (LLama 2 70B is worse than CodeLlama 34b in coding tasks) in our evaluations. The following table presents the results from our extended evaluation on the full context Arcade benchmark:

| Model | Code LLM (no spec.) | CodeLlama-Python | CodeLlama-instruct |  Ours (synthetic f.t. w. I/O Summary) |
|-----|-----|-----|------|------|
| pass@20 	| 37.5 | 36.2 | 36.0 | 55.47 |

These results indicate that the base Code LLM, used as the starting point for GIFT4Code, is indeed a competitive model. Additionally, the comparable performance of the 15B StarCoder, 34B CodeLlama and the vanilla 62B Code LLM showed that the Arcade benchmark is very challenging, which could not be solved by just increasing the number of parameters, highlighting the unique value GIFT4Code brings to the data science domain.

Reviewer comment (Reproducibility):
It is not clear how generalizable Gift4Code is. The implementation and training data of this paper are not publicly available, making it difficult to estimate the generalizability of the method, i.e., whether the improvement is restricted to the unknown/anonymous code LLM or if it will work similarly effectively when applied to other pre-trained code LLMs. The authors are asked to conduct similar instruct-tuning using open-source code LLMs, such as code llama-7B, 13B, and 34B.

Author response:
> How generalizable is GIFT4Code across different code LLMs?

Given limited computational resources, in this paper we only focus on improving one code LLM to follow NL instructions with I/O specifications, in line with existing works on fine-tuning LLMs using synthetic data [1,2]. We chose the base LLM that has the most promising performance in the data science domain. Nevertheless, we totally agree with you that the best strategy is to verify fine-tuning approaches across different LLMs. We leave this as important future work.

[1]: Wang, Yizhong et al. “Self-Instruct: Aligning Language Models with Self-Generated Instructions.” ACL (2022). 

[2]: Taori, Rohan et al. “Alpaca: A Strong, Replicable Instruction-Following Model.”

Reviewer comment (Experiments):
In the realistic development scenario, the expectation is that only the human intent is given. In such a case, the ground-truth code is unknown, so the summary of execution results relying on the "generalist" LLM is not possible. In this case, where is the I/O summary from? Will the user have to further indicate the specified I/O beyond the high-level intent? Is such a requirement practical?

Author response:
> Real-world setup of GIFT4Code?

After model deployment, we anticipate that developers will provide I/O specifications in their prompts, a similar assumption used in [1]. GIFT4Code is designed to tackle a prevalent challenge: even when complex I/O specifications are provided, models often fail to align with them, leading to inaccuracies in code generation. This is a significant issue in practice, where developers rely on the model to follow closely to the provided specifications for correct code generation. 

We conducted a preliminary experiment where the code LLM first generated an I/O summary and then the code solution. The pass@20 is ~49.3% which is lower than the results obtained with high quality but noisy specifications generated by the “generalist LLM” (55.5%). Acknowledging that real-world scenarios might not always offer explicit I/O summary, extending GIFT4Code to adapt to such conditions remains a key area for future work.

[1]: Shi, Kensen et al. “TF-Coder: Program Synthesis for Tensor Manipulations.” TOPLAS (2022)

Reviewer comment (Reproducibility):
If releasing the artifact is not possible, could the authors provide a detailed description of the training process as a reproducibility statement?

Author response:
> Lack of details in the model training, configuration, and architecture

We appreciate your concerns regarding the reproducibility of our work. Here are some more details on the LLMs. The code LLM is a state-of-the-art large code language model fine-tuned for Python data science applications. It is a decoder-only Transformer with 62B parameters. It has 64 layers, 8192 hidden units, 32 attention heads, and uses RoPE embeddings with a 256k token SentencePiece vocabulary. The model was first pre-trained on a collection of 1.3T tokens of web documents and github code data, and was then fine-tuned on a disjoint set of 64B Python code tokens together with 10B tokens from Python Jupyter notebooks. The fine-tuning process, including the GIFT4Code procedure, can be conducted through the model tuning service on Google cloud Vertex AI. The learning rate of GIFT4Code fine-tuning is 1e-5 and other training details (steps, batch size) can be found in the experiment section.

The base model is publicly available as an API, but was only privately accessible at the time of submission. The model details are publicly available in \citet{anonymous}. We will make sure to report more model information in the final version and deanonymize the LLM citations.