Paper: Coarse-Tuning Models of Code with Reinforcement Learning Feedback (ICLR-2024)

Reviewer comment (Experiments):
The proposed approach is limited to larger dataset due to the fact that CODENETJAVA does not consider dependencies on user-defined packages or libraries.

Author response:
The use of CodeNetJava, which excludes user-defined packages, is a deliberate choice to streamline static analysis during training and facilitate rapid prototyping of RLCF. This dataset choice doesn't limit the applicability of RLCF, as the framework is equally applicable on larger datasets containing user-defined packages. For such datasets, incremental compiler tools like Bazel can be leveraged as they efficiently rebuild modified program parts considered during RLCF-based model training.

Reviewer comment (Experiments):
The paper only evaluates the proposed approach on two specific tasks for Java, which may not be representative of other programming languages or models.

Author response:
The RLCF framework is designed to be applicable to all languages and models of various sizes. While our experiments focus on Java and a range of model sizes (from 350M to 1.5B), the extension of RLCF to other languages and models is a straightforward task, and we acknowledge this as a potential avenue for future work.

Reviewer comment (Experiments):
This work does not provide a comparison of the proposed approach with existing state-of-the-art LLMs like GPT3 or GPT4.

Author response:
GPT models, notably GPT-3 and GPT-4, are significantly larger than the models examined in this paper. Unfortunately, their usage comes with associated computational costs, which proved difficult to procure in an academic setting. We can only ask that the reviewer evaluate our proposal on the models that we were able to run.

Reviewer comment (Experiments):
How this approach can work with existing pre-trained code-specific LLMs is missing.

Author response:
The selected CodeGen (Decoder only) and CodeT5 (Encoder-Decoder) models are code-specific language models (LLMs), and our study demonstrates improvements in generated code with RLCF applied to these models. The RLCF framework is adaptable and can be similarly applied to other existing code-specific LLMs.

Reviewer comment (Experiments):
Is the proposed evaluation and experiments representative enough for other programming languages or models?

Author response:
The RLCF framework is designed to be applicable to all languages and models of various sizes. While our experiments focus on Java and a range of model sizes (from 350M to 1.5B), the extension of RLCF to other languages and models is a straightforward task, and we acknowledge this as a potential avenue for future work.

Reviewer comment (Experiments):
How can this approach work with existing pre-trained code-specific LLMs is missing?

Author response:
The selected CodeGen (Decoder only) and CodeT5 (Encoder-Decoder) models are code-specific language models (LLMs), and our study demonstrates improvements in generated code with RLCF applied to these models. The RLCF framework is adaptable and can be similarly applied to other existing code-specific LLMs.

Reviewer comment (Reproducibility):
From the perspective of practical implementation in developer tools, the requirement to utilize both a compiler and an additional discriminator model could present significant challenges.

Author response:
***"From the perspective of practical implementation in developer tools the requirement to utilize both a compiler and an additional discriminator model could present significant challenges."***: The discriminator and compiler are solely utilized in the model-training phase. In practice during deployment in developer tools, only the RLCF-tuned model would be used, mirroring the deployment setup of tools like Codex.

Reviewer comment (Experiments):
The study compares the base pre-trained CodeGen model with the version enhanced by RLCF. Would it be feasible to include a comparison with a supervised fine-tuned variant of the CodeGen? For instance, we could create a collection of "gold standard" examples that both compile successfully and are preferred by the discriminator for SFT. This could separate the improvement from the coarse tuning technique itself vs better quality data.

Author response:
***"The study compares the base pre-trained CodeGen … vs better quality data"***: We've implemented a 'Mono' Baseline, a supervised variant trained on gold-standard responses from the dataset. This baseline uses samples that successfully compile and align with prompts, providing a comparison to gauge the impact of the RLCF technique on top of high-quality data.

Reviewer comment (Experiments):
The process of generalizing this approach across various programming languages, and different compilers might represent a challenge. If possible, add more tests or discuss it.

Author response:
***"The process of generalizing this approach across various programming languages, and different compilers might represent a challenge. If possible, add more tests or discuss it."***: The RLCF framework is designed for strongly typed languages with standard compilers, such as Java. Extending it to dynamically typed languages like Python is more challenging as it requires robust static analysis tools. This is why we prototyped the method using Java.

Reviewer comment (Experiments):
Additionally, a comparison with state-of-the-art reinforcement learning techniques that use feedback would be valuable. Utilizing a compiler to generate binary rewards could help assemble a training dataset suitable for a conventional RLHF or RLAIF frameworks, providing a stronger baseline to evaluate the effectiveness of RLCF.

Author response:
***"Additionally, a comparison with state-of-the-art reinforcement learning techniques … effectiveness of RLCF."***: We have compared RLCF against CodeRL, a strong RL baseline, showcasing improvements with RLCF. CodeRL itself relies on a dataset assembled with binary rewards from the compiler. Further details are available in Appendix Sec 8.6 for reference.