Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2024)

Abstract: Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training (WRAP) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as “like Wikipedia” or in “question-answer format” to jointly pre-train LLMs on real and synthetic rephrases. First, we show that using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training by ~3x. At the same pre-training compute budget, it improves perplexity by more than 50% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data has higher utility than just real data because it (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher ‘quality’ than web-scraped data.

## introduction large language model (llm) pre-training has been largely democratized and open-sourced, allowing various academic labs, and industries to pre-train custom llms. yet, a key differentiator between these models is the composition and size of the data used to train them. data curation strategies * equal contribution † work done during internship at apple are required to filter out scrapes of the web that are unstructured and/or poorly phrased @xcite . while some of these strategies have been made public @xcite @xcite , most state-of-the-art data curation techniques are unknown to the research community, and only anecdotal evidence remains. research on data curation requires multiple rounds of re-training, making it an expensive endeavour to document techniques that lead to practical improvements. on the other hand, scaling laws for language models (such as chinchilla scaling laws @xcite ) show that with increasing model sizes, we should also increase both the training compute and data size linearly. this is infeasible because (a) high-quality data is limited @xcite , and repeating for even a small number of epochs (4 or more) results in diminishing returns or overfitting @xcite @xcite ; and (b) pre-training for such long durations is prohibitively expensive. meanwhile, the use of synthetic data has gained prominence in the paradigm of aligning pre-trained llms via instruction fine-tuning, rlhf @xcite , and instruction backtranslation @xcite . recently, in the context of pretraining, synthetic data was used to generate datasets such as tiny stories @xcite and textbook quality synthetic data @xcite . these were used to train smaller language models (like the phi model family) that were as performant as larger language models on certain tasks. however, their data generation process stays largely opaque, and prohibitively expensive, requiring prompting a gpt-3.5 model for generating billions of tokens. additionally, such data generation can create "knowledge bias" by generating data pertaining to tasks that we want to perform well on. while synthetic data has shown promise, it is unclear if this is be-cause of its higher quality, or strategic topic selection @xcite . in this work, we propose web rephrase augmented pre-training (wrap)-that attempts to bridge three important challenges stemming from the ambiguity around data curation-(i) what data should you pre-train on? (ii) how can you pre-train with limited data? (iii) how can you pretrain computationally efficiently? in particular, we show that re-phrasing documents on the web using an off-the-shelf medium size llm allows models to learn much more efficiently than learning from raw text on the web, and accounts for performance gains on out of distribution datasets that can not be offset with additional web data. our proposed method involves using a pre-trained off-the-shelf llm to re-phrase documents from a web corpus into different styles (figure 1a ). in our work, we tackle two important challenges faced during synthetic data curation in the works of @xcite -generation cost, and data bias-by rephrasing articles on the web. (i) wrap allows for using an open source, and much smaller llm (1.8b/7b v/s gpt3.5) to rephrase unstructured and poorly phrased documents in different styles, since it does not rely on the llm as a knowledge bank. (ii) thanks to the information maintaining nature of rephrasing, we are able to leverage the natural diversity of the web rather than relying on an llm for information, which may be prone to factual errors, and data biases. wrap shows that the "style" alone can result in large gains in pre-training efficiency, and downstream performance. using wrap on the c4, we evaluate model performance on 13 different zero-shot tasks, and 21 different language modeling domains of the pile, and find that pre-training llms with synthetic data allows us to train equivalent models with 5x lesser data, or 3x lesser compute. in fact, our models, also outperform the recent tinyllama models that were trained for 3 trillion tokens (10x data and compute) across several zero-shot q/a tasks. we further observe a reduction in perplexity by ∼ 50% on the pile, and note that our 350m parameter model trained on combinations of real and synthetic rephrases on just 15% of the entire c4 corpus, outperforms pre-training a 1.3b parameter on the entire c4. finally, we conduct an analysis on the potential of data leakage, properties of synthetic data styles, and how to combine synthetic data for improving wrap based llm pre-training. dataset selection selecting high quality data for pre-training llms remains an active, high-impact, yet understudied area of research. for instance, gpt-2 model was pre-trained on all outbound links from reddit, a social media platform, which received at least 3 karma @xcite . this was used as a heuristic indicator that the document may be interesting, educational, or just funny. follow-up works have used other heuristics such as prioritizing documents that resemble wikipedia @xcite . @xcite used multiple heuristic filters to remove documents, such as the absence of certain stopwords, length of the document, percentage of alphabetic characters, mean word length, symbol-to-word ratio, percentage of lines starting with a bullet point, or ending with an ellipsis etc. their work highlights the intricacies of filtering out text data. an alternative paradigm for building better datasets for training is to distill high-quality datasets. xie et al. (2023) proposed a method, doremi, to select the best data mixture for pre-training language models by reweighting data from various domains. concurrently, @xcite showed that de-duplicating pre-training data can improve pretraining efficiency. recently several methods were proposed for automatic filtering of low-quality data for faster fine-tuning of llms @xcite @xcite . simultaneously, in the realm of image-language models such as clip @xcite , the datacomp benchmark @xcite and recent entries @xcite developed approaches at filtering out low-quality subsets from pre-training datasets like laion @xcite , or the common crawl. @xcite showed that a synthetically generated dataset in the form of stories that toddlers can understand allows training a small language model that can generate coherent sentences. @xcite showed that textbook quality (synthetic) data alone helps models achieve stateof-the-art performance on reasoning and coding tasks. similar approaches are used in concurrent work for enhancing coding and mathematical reasoning abilities while finetuning @xcite . @xcite show that training on synthetic data can actually be harmful for model performance, especially when we do multiple rounds of pre-training an llm and then training the next llm on data generated by the previous one. on the other hand, some other works have shown that such a strategy can actually be useful. @xcite and @xcite discuss how a model can generate instruction data and then fine-tune on its own generated data to improve performance. @xcite discuss how such repeated cycles of synthetic data can help train a very small paraphrase model that outperforms gpt-3. ## wrap: web rephrase augmented pretraining generating synthetic data using an off-the-shelf language model can be both computationally expensive and operationally challenging. prior approaches to generating synthetic textbook quality data using llms @xcite required (1) a language model that contains sufficient world knowledge to generate articles worth training on, thereby increasing generation cost; ## perplexity evaluation we evaluate the perplexity of the pre-trained model on the validation set of multiple out-of-distribution datasets. all models are either trained on the c4 dataset @xcite , or a particular stylistic average wiki owt2 books2 books arxiv stackx github math hnews enron pubmed-c ubuntu pubmed-a freelaw nih uspto pg-19 phil youtube opensubs cc 0 20 40 60 perplexity c4-170b c4-85b c4+qa-85b rephrase of the same. all the evaluations are done on 21 sub-domains of the pile @xcite . these subsets are created from the first 10,000 documents from each domain of the pile dataset. we then evaluate the perplexity of the model on these subsets. additional evaluation details are provided in appendix d. it is important to note that we evaluate perplexities on the pile instead of c4. training on multiple distributions of text (synthetic and real web) does comes at a marginal cost of less than 1 perplexity on the c4 validation set. to understand our choice of evaluation, and why we observe this perplexity increase, we note that training over the c4 corpus corresponds to minimizing the objective solving for equation 2 does not minimize the risk over c4-only, and hence the small drop in performance between θ c4 and θ wrap on the c4. for meaningfully comparing models trained on the c4 and on its synthetic rephrases, we evaluate their generalization capability on 21 different domains of the pile @xcite . results for each domain are presented in figure 2 . ## zero-shot tasks we now evaluate our pre-trained language models on various zero-shot question answering (qa) benchmarks using the llm evaluation harness foot_0 @xcite . ## rephrase examples from c4 corpus we show one sample document from the c4 corpus below, and then its corresponding rephrase in medium and q/a style when using mistral-7b as the rephraser. more such examples are provided in appendix j. for the medium style rephrasing, we ask the llm to rewrite the sentence in high quality english such as wikipedia, whereas for the q/a style rephrase, we ask the llm to rewrite the document in conversational question-answering format. the detailed prompts are in appendix i. ## conclusion strong language models are being pre-trained on combinations of real and synthetic data. using synthetic data enables baking in desirable attributes such as fairness, bias, and style (like instruction following) directly into the data, eliminating the need to adjust the training algorithm specifically. this offers an alternative approach to aligning language models to human values. the recent uptick in interest around synthetic data, especially for instruction-tuning language models, is noteworthy, with concurrent researchers also leveraging it for pre-training. as we transition into this paradigm, understanding the properties of the data fed to our models is paramount. this paper aims to be a comprehensive guide on employing different synthetic style data in llm pre-training. we delve into its significance from two vantage points: (1) in scenarios with scarce high-quality data, synthetic rephrases offer more value than mere repetition of existing data;