Paper: Textbooks Are All You Need (ICLR-2024)

Reviewer comment (Experiments):
Can you please briefly introduce the training data and method used by CodeGen-Mono, Replit and StarCoder? Will be helpful in understanding how Phi-1 is different and perhaps will further highlight the important of data quality.

Author response:
Thank you for raising the point. For Codegen-Mono we don’t know the exact dataset that was used in the training, but their paper states that they use combination of Python source code data from Github and NLP data from The Pile. Both these datasets are large corpuses of web data without additional filtering. StartCoder and Replit were trained on the deduplicated subset of The Stack (which in turn has github source code cleaned of PII and permissive licenses among other things). Both these models also use multiple code languages in pretraining. These models were trained on a total of 1T and 500B tokens, respectively.  

In comparison Phi-1, (a) we start with only the python subset of The Stack and stackoverflow, (b) but uses filtering to reduce the size to 20-25% of the original datasets (6B unique tokens), and (c) strategically add targetted synthetic data in pretraining and finetuning. We train on 7B unique tokens for slightly less than 8 epochs. Of the choices we made, the first choice of using python-only was not strategical and made for restricting the scope so we can do more controlled experiments. The other two choices were the new contributions in this work.

Reviewer comment (Experiments):
As mentioned in Sec 3.2, fine-tuning on coding exercises makes unrelated tasks easier to distill from pretraining. If I'm understanding it correctly, during pretraining, languages other than Python are used. Is it possible to further support this claim by evaluating on non-Python coding tasks?

Author response:
Unfortunately, our pretraining data had only python code snippets, but this would be a good experiment to expand on our model for multilingual code. This could certainly help distil information from other languages, although one consideration experimentally is that if finetuning is fully in python the model might be overfit to the python syntax. To avoid this, the right approach might be to mix in python CodeExercises with small amounts of filtered dataset from other languages – this way we can see the benefits of training with python-CodeExercises on other languages without being confounded by the syntax overfitting.

Reviewer comment (Evaluation):
It was mentioned that Phi-1 has emergent abilities that were not observed in Phi-1-base. In Sec 3, examples were used to support this claim. Is it possible to evaluate this more systematically and qualitatively?

Author response:
The main surprising emergent ability we found was that the model got better at API usage of popular apis even if the finetuning dataset did not have those API’s. Unfortunately, there are no standard benchmarks we could find for evaluating API usage, so we show more qualitative examples of the correct usage.

Reviewer comment (Experiments):
The authors are motivated to include high-quality data (i.e., textbook). However, they also use the data generated by gpt-3.5, which may have a lot noises. How to justify the motivation because it seems contradictory to have both high-quality and low-quality data.

Author response:
While GPT-3.5 is far from perfect and has noises, the comparison point here are random code/text snippets from web which even after filtering, which typically have much higher levels of non-educational content. We find that with the right prompting, especially for simple coding problems the quality of GPT-3.5 generated data has far more educational value compared to a typical random text from Github repos (which form the large part of web data) or StackOverflow. The harder issue to tackle with using Synthetic data is that the generated text might not have sufficient coverage of rarer and more complex use-cases. This is why we use the mix of web-filtered data with synthetic data targeted to address specific shortcomings we observe in the webdata. We also note that our prompts were tailored to the generating model (GPT3.5)’s known strengths and weaknesses. Taking the example of synthetic textbooks, these were designed to cover the entirety of beginner programming topics, but at the same time, the topics selected were fine-grained enough that GPT-3.5 can reliably generate high accuracy text. We ensured diversity through variations in format and target audience. Of course, if we use a more powerful GPT-4 model for our generations the quality of models will very likely improve further, but generating data at our scale using GPT-4 is significantly more expensive and slower (due to access limits).

Reviewer comment (Presentation):
In the caption of Figure 1, it is not light / dark green. maybe light / dark blue?

Author response:
Thank you for pointing this out, this was a typo (we forgot to edit text after changing the figure) and we will fix it.

Reviewer comment (Experiments):
In section 2, the authors are motivated by: We conjecture that language models would benefit from a training set that has the same qualities as a good “textbook”: it should be clear, self-contained, instructive, and balanced. It may be too intuitive without any justification. What about other high-quality data in addition to textbook? For example, legal documents, government document?

Author response:
We certainly considered official sources of coding textbook data, there are two issues we encountered: (a) much of such high-quality material including real textbooks are copyrighted material, (b) the size of the datasets we managed to obtain while looking for such sources was very small (<<100MB) that it did not end up making much of a difference.

Reviewer comment (Experiments):
In Section 2.1, the authors rely on the GPT-4 as an annotators to filter the data. Why can we rely on it? I am aware of some studies that using GPT-4 as an annotator. However, the ability of GPT-4 on this specific task is unknown. Justifications are needed. For example, you may want to show the comparison between the data that GPT-4 thinks high-quality v.s. low-quality.

Author response:
Indeed this is an imprecise measure. We rely on it only after many iterations of manual quality checks and prompt refinements – we had small batches of data scored with GPT-4 and human labelers and look at correlations and iterate on the prompts based on feedback. We provide a typical example of high and low educational value samples from The Stack in Page 4.

Reviewer comment (Reproducibility):
The paper claims that meticulously curated data, combined with generating synthetic training data, can train smaller models that surpass larger ones. However, there are significant gaps regarding the creation of the training data, as the authors deliberately omit certain details and refer to other papers that have taken a similar approach (as mentioned in footnote 1 on page 2). While withholding such details is at the researcher's discretion, it makes it challenging to assess the paper as a standalone piece and to be certain that the methodology doesn't introduce any data leaks potentially favoring the results.

Author response:
**Q. However, there are significant gaps regarding the creation of the training data. … assessing the paper as a standalone piece makes it challenging to be certain that the authors' methodology doesn't introduce any data leaks potentially favoring their results.**

We acknowledge this as a limitation of our work for academic publication. We have added as much detail in the paper as we thought were appropriate to balance between proprietary interests and openness to research community. We additionally release our model artifact for further research, exploration, and independent evaluation. While this is not ideal or 100% transparent, we nevertheless believe that our paper captures important aspects of LLM training and is important for the research community. As noted in the paper it is also not unprecedented in our research community where highly influential papers have not disclosed dataset details (e.g. Minerva which also attribute their main contribution to be data).

Reviewer comment (Experiments):
Although the authors explore leakage in the finetuning data, the same scrutiny is not applied to their synthetic data.

Author response:
**Q. Although they explore leakage in the finetuning data, the same scrutiny isn't applied to their synthetic data.**

We were able to do a thorough decontamination analysis for HumanEval on our CodeExercises dataset as both datasets were of the same type (short python functions) and are self-contained code snippets – this meant that we can devise robust versions of similarities beyond string matching. We note that many decontamination procedures in the literature (including The Stack dataset) resort to string-matching or min-hash matching against a test set which we show is not very robust (e.g., our analysis picked up similarities that N-gram analysis did not).   

On the other hand, our synthetic textbook is a fundamentally different NLP-heavy dataset. The code snippets in the text are often very short and used mostly for illustrative purposes: e.g, exercises or usage specific functions or structures like print and if-then, class definition etc. These are not directly comparable to HumanEval questions which have specific function level algorithmic style. Thus, we did not find this dataset suitable for in-depth ast-style analysis.

Reviewer comment (Evaluation):
The approach taken to assess leakage against the finetuning data is somewhat nebulous. In their n-gram overlap analysis, they offer no rationale for their choice of 13-grams to gauge overlap, quickly concluding that “n-grams are not refined enough to find similar code snippets.” Subsequently, they switch to embeddings, stating, “We observe that the embedding distance effectively captures code pairs,” without substantiating this claim.

Author response:
**Q. The approach taken to assess leakage against the finetuning data is somewhat nebulous. In their n-gram overlap analysis, they offer no rationale for their choice of 13-grams to gauge overlap, quickly concluding that “n-grams are not refined enough to find similar code snippets”. Subsequently, they switch to embeddings, commenting, “We observe that the embedding distance effectively captures code pairs,” without substantiating this claim.**
and 
**What verification, if any, was performed to rule out that the synthetic pre-training data generated is not highly similar to the existing samples in HumanEval?**

Many standard datasets in literature perform decontamination by using exact substring matches or N-grams matches to remove test set from training set: exact substring (extreme version of N-gram with N=len(human-eval-question)) was used in The Stack dataset which is the defacto standard dataset for code models; one of the largest NLP datasets DOLMA (https://blog.allenai.org/dolma-3-trillion-tokens-open-llm-corpus-9a0ff4b8da64) uses 13-gram like ours. In our 13-gram detection we did not detect any true-positive contamination in over 100k code snippets of CodeExercises (we had 4 false positives and all else were negative). Given the widespread use of N-grams for decontamination, we do not consider our conclusion unfounded. 

That said since we were ourselves not satisfied with this common procedure which is why we go beyond this standard practice and use other techniques for determining code-similarity, we show how code-snippets that are similar in principle are not detected by N-gram but are captured by our metrics.

Reviewer comment (Evaluation):
The authors assert that phi-1 consumed less compute for training but overlook the computational resources expended in creating their training data and, more importantly, the compute required to train the foundational LLMs. While this does not directly undermine the premise that a lesser-compute model can theoretically surpass larger counterparts, it tempers its implications given that generating such data on a large scale necessitates the distillation of more resource-intensive models.

Author response:
**Q. Another point of contention is the authors' assertion that phi-1 consumed less compute for training. They overlook the computational resources expended in creating their training data, and more importantly, the compute required to train the foundational LLMs.** 

This is a very fair point that we did not think about and thank you for pointing it out. We will clarify the hidden costs of our model required for creating “good quality” training data. Thank you also for noting that this alone does not undermine the results in our work. We wanted to couple additional points on this note: (a) it would be fairer to consider the amortized cost of building the foundation models (GPT4, GPT3.5) but these estimates are hard to get before their widespread use; (b) even without GPT-3.5 generation (no Synthetic textbooks, and no finetuning), our 350M parameter model trained on filtered data alone achieves ~18% performance on HumanEval (in comparison, same model trained with unfiltered data for 3x longer only achieves 12%) – we did not run this combination on 1.3B scale that our other base model is at. In this case, the only additional overhead our classifier and small number of GPT-4 annotations for quality.

Reviewer comment (Presentation):
In the paper, all references are given using \citet, while the expected style calls for using \citep whenever the authors or the publication are not included in the sentence. This should be corrected following the style guidelines (https://iclr.cc/Conferences/2023/CallForPapers).

Author response:
**Q. Comments on presentation.**
Thank you for the many suggestions on improving the presentation. We will fix all of them.  
- Citation style: We will fix the citation style

Reviewer comment (Experiments):
On page 2, the paper states that it confirms the hypothesis that the number of parameters plays the key role in emergence. While there is much work showing the role of parameter count in performance, there is also notable evidence against the emergence hypothesis. Since this work only contains two model sizes, the language “confirm the hypothesis” is too bold, as there is no way to differentiate a sudden jump from a smooth transition over a simple line.

Author response:
**Q. Comments on presentation.**
Thank you for the many suggestions on improving the presentation. We will fix all of them.  
- Discussion of emergence: We agree that with two model sizes our statement is strong, and will tone down to say emergence could be one possibily on why we see surprising behavior at 1.3B that were not noticeable in 350M.

Reviewer comment (Evaluation):
On page 6 §3.2, the authors claim “we demonstrate that finetuning on CodeExercises unexpectedly improves the model’s ability to use external libraries…”. This phrasing is not supported by the evidence given in the paper, as the evaluation is qualitative with a single example. Moreover, from the example, it is clear that the model saw PyGame in the pretraining data and even phi-1-base knew how to use it. Instruction tuning is meant to help the model generalize from stored knowledge, so there is nothing unexpected about this improvement.

Author response:
**Q. Comments on presentation.**
Thank you for the many suggestions on improving the presentation. We will fix all of them.  
- **Improvement from finetuning**: Here we disagree with the reviewer partially – we agree that our exploration of improved API use is qualitative and will remove any quantitative/rigorous statements about it. But we disagree with the reviewer that the finding that our specific finetuning improves API usage performance is not surprising. Our finetuning is not what would be commonly considered “instruction tuning”. If anything since CodeExercises only contained algorithmic codes that DO NOT use complex APIs, our prior expectation was that the base model performance would degrade after finetuning and we found it surprising that it indeed improved significantly. Also while we provide only one example in the main paper, we show additional examples in the appendix.

Reviewer comment (Novelty):
The paper has low novelty, and the importance of data quality in LLMs is well known.

Author response:
On this point, we respectfully disagree with the reviewer about the lack of novelty in our paper. While the philosophy of good quality training data being good for learning is not new, the predominant trend in LLMs has been on scaling-up data and model sizes, as evidenced by numerous papers on scaling laws. To our knowledge, our paper is among the first to provide strong and compelling evidence on the effectiveness of high-quality data to achieve state-of-the-art performance in LLMs. The novelty in our work also lies in specific approaches we took to generate and use "textbook quality" data. Our paper contributes uniquely in several ways: 
- Targeted synthetic data: We show the effective usage of targeted synthetic data to address specific gaps in existing data sources: e.g. our synthetic textbooks provide useful signals for the model by enhancing signal from natural language text that explains simple algorithms in detail. 
- Filtering methodology: We used a combination of GPT-4 (frontier LLM), a small LM classifier, and human-in-the-loop for textbook-quality data filtering. Such methodologies incorporating frontier LLMs strategically in the loop are in the early stages of experimentation and our paper contributes to it. We believe identifying best practices in this area is significant for future models. 
- Overall, our model achieves top-tier performance on algorithmic code generation (next only to latest GPT models) despite being 10x-100x smaller in size compared to the other top models. This provides strong evidence that selecting high quality data in creative ways is key to unlocking efficient learning.

Reviewer comment (Novelty):
How do the results of synthetic data, and using textbooks generalize to other domains beyond code? Does this approach advance the scientific understanding of training language models for code generation?

Author response:
Our paper focuses on code generation as a proof of concept to demonstrate the benefits of using textbook-quality data in training LLMs. This serves as a compelling standalone use-case and a representative example for other tasks that require reasoning-based skills. We are aware of similar follow up attempts in more general language understanding and reasoning tasks.