Paper: OctoPack: Instruction Tuning Code Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
I wonder if instruction tuning the model on more data from CommitPackFT can lead to even better performance.

Author response:
As mentioned above, we found that the models converge extremely quickly. In fact, the 5,000 samples are not even entirely seen during our instruction tuning of OctoCoder. We think that this is due to most of the capabilities being learned during pre-training, while instruction tuning mostly serves the purpose of teaching the model the expected input and output format.

Reviewer comment (Experiments):
The discussion around CommitPack leads with the rather enormous size of the initial dataset (~4TB). While this is accurate, the version used in this work is just 0.05% this size (2GB). This work offers no validation or experiments involving the original set. This both creates the impression that the 4TB set is likely to be very noisy, and suggests that this work should really primarily emphasize the 2GB portion as far as concrete contributions of this paper are concerned. That would involve amending text like in the contributions on P2 and in the abstract, which don't mention the size of the subset that was actually used at all. This also relates to the "orders of magnitude more" comment on P9, which seems quite inconsequential.

Author response:
We do use the entirety of CommitPack for pretraining in `Appendix F`. We should have made this clearer, so we added a pointer to this appendix section in `Section 2. CommitPack: Code Instruction Data`. Given this, we would leave our claims as is, but please let us know if you would still adjust them.

Reviewer comment (Experiments):
The work makes a rather strong argument that building on HumanEval is a positive since it is so commonly filtered from training data. This argument is challenging to back up, given the absence of insight into how many LLMs are trained and anecdotal observations that pretrained models perform worse on new coding problems (e.g. from CodeContests). A more moderated discussion of this choice, acknowledging the use of HumanEval as a potential limitation, is suggested.

Author response:
Thanks for pointing this out. Indeed, inventing entirely new problems would have been better than building on HumanEval. We have rephrased the end of Section 3 to be more moderate and acknowledge this limitation.

Reviewer comment (Experiments):
Appendix C notes that inputs were filtered to just 768 tokens in length for the complete before-code + message + after-code series. This is a very low limit by modern language modeling standards. Tab. 8 reinforces that this leads to very short programs compared to the natural distribution of commits. The motivation for these limits is unclear, and their impact on the dataset's usefulness for downstream use-cases—where programs are rarely as short as those in HumanEval—needs clarification.

Author response:
Great point. The reason for this limit was that a large fraction of the code after is usually the same as the code before. Thus, much of the finetuning would be wasted on teaching the model to copy code from the before to the after. To increase the signal per token (and thus have the model learn faster) and avoid having a model that always repeats user input, we only finetune on relatively short commit pairs. We explain this in Appendix D - we have added some more details to that explanation.

Reviewer comment (Presentation):
The center panel in Fig. 3 might benefit from a visual improvement. Without reading the text, it is unclear what is happening; as presented, it looks like the model is prompted with the code, predicts text, then is prompted with more text and predicts code again, all in one input sample. Consider inserting a blank space between the two tasks (before the second model input) and moving the arrow to connect the two halves.

Author response:
Thanks, we have updated the plot according to your suggestion.

Reviewer comment (Experiments):
Sec. 3, P5: Clarification is needed on what experiments were conducted to confirm that GPT-4's accuracy only varies by 2% when sampling 1 sample vs. 20. The variation could be greater on some tasks. It might be worth sampling 2–5 samples to balance cost and precision.

Author response:
Good point, we have rephrased that sentence to `For GPT-4, we generate n=1 samples. Using n=1 instead of n=20 for GPT-4 only changed scores from 75.0% to 75.2% pass@1 on HumanEvalSynthesize Python while providing 20x cost savings.` which hopefully clarifies this?

Reviewer comment (Evaluation):
It would help the work to also report BLEU/METEOR scores on the explanation generation task. These might offer a complementary view to the proposed automated metric, which could be somewhat lossy.

Author response:
As suggested, we have added a comparison to BLEU and METEOR in Appendix N. Note that for BLEU we assume that the docstring is a ground-truth explanation which can be a problematic assumption as docstrings are not necessarily meant to be explanations.

Reviewer comment (Experiments):
A few odd notes from the appendix: why was "can’t you see i’m updating the time?" used as a prominent commit message filter (Tab. 5)? This doesn't seem to be a common phrase. Also, why was only data up till 2016 used, given there are archives with more recent GitHub data?

Author response:
Surprisingly, “can’t you see I’m updating the time?” is one of the most common commit messages in our GitHub dump. You can check this blog post where it also appears as the 6th most common message: https://dev.classmethod.jp/articles/my-favorite-bigquery-dataset-newshiro/ or this one from Google: https://codelabs.developers.google.com/codelabs/cloud-bigquery-csharp#5. We’re not sure why this is the case, but decided to remove those samples, as it is not a useful commit message. The data we used unfortunately only contained data until 2016 (https://github.blog/2016-06-29-making-open-source-data-more-available/). We are working on releasing a new commit dataset with more recent data.

Reviewer comment (Evaluation):
Why do OCTOCODER and OCTOGEEX perform best among permissive models?

Author response:
The difference is the pre-trained base model and some minor hyperparameters. While OctoCoder is initialized from StarCoder, OctoGeeX is initialized from CodeGeeX2. There are tiny differences in the hyperparameters during finetuning, which are detailed in `Appendix O: Hyperparameters`. Indeed OctoGeeX outperforms OctoCoder on Go and Rust for HumanEvalExplain. The base model (CodeGeeX2) also performs better than OctoCoder’s base model (StarCoder) for Go and Rust (https://github.com/THUDM/CodeGeeX2/blob/main/README_EN.md) which could be the reason for this.

Reviewer comment (Evaluation):
Why does OCTOGEEX have better performance in Go and Rust languages compared to OCTOCODER?

Author response:
The difference is the pre-trained base model and some minor hyperparameters. While OctoCoder is initialized from StarCoder, OctoGeeX is initialized from CodeGeeX2. There are tiny differences in the hyperparameters during finetuning, which are detailed in `Appendix O: Hyperparameters`. Indeed OctoGeeX outperforms OctoCoder on Go and Rust for HumanEvalExplain. The base model (CodeGeeX2) also performs better than OctoCoder’s base model (StarCoder) for Go and Rust (https://github.com/THUDM/CodeGeeX2/blob/main/README_EN.md) which could be the reason for this.