Instruction Pre-Training: Language Models are Supervised Multitask Learners (2024)

Abstract: Unsupervised multitask pre-training has been the critical method behind the recent success of language models (LMs). However, supervised multitask learning still holds significant promise, as scaling it in the post-training stage trends towards better generalization. In this paper, we explore supervised multitask pre-training by proposing Instruction Pre-training, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train LMs. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction response pairs covering 40+ task categories to verify the effectiveness of Instruction Pre-training. In pre-training from scratch, Instruction Pre-training not only consistently enhances pre-trained base models but also benefits more from further instruction tuning. In continual pre-training, Instruction Pre-training enables Llama3-8B to be comparable to or even outperform Llama3-70B. Our model, code, and data are available at https://github.com/microsoft/LMOps.

## introduction on the path towards general artificial intelligence, multitask learning @xcite emerges as a promising approach. however, scaling supervised multitask learning to the necessary degree is very challenging. this motivates @xcite to explore unsupervised multitask learning: pre-training on raw corpora through causal language modeling, which facilitates scaling up training data. over time, unsupervised multitask learning has evolved into the standard approach for pre-training language models (lms) @xcite , which is referred to as vanilla pre-training in this paper. despite the success of unsupervised approaches, supervised multitask learning still holds significant promise. instruction tuning @xcite , which fine-tunes pre-trained models using diverse tasks framed through natural language instructions, significantly enhances task generalization @xcite , re-emphasizing the value of supervised multitask learning. in this paper, we introduce instruction pre-training to explore supervised multitask learning for pre-training. as shown in figure 1 , instead of directly pre-training on raw corpora, instruction pre-training augments each raw text with a set of instruction-response pairs foot_0 generated by an instruction synthesizer, and then pre-trains lms using the augmented corpora. these pairs are synthesized based on the content of massive raw corpora, ensuring high knowledge coverage and correctness. therefore, we can scale up task synthesis with great diversity and quality @xcite . to develop the instruction synthesizer, we convert a wide range of existing datasets into our required format: each example consists of a set of instruction-response pairs and a piece of raw text that these pairs condition on. using this data collection, we fine-tune a language model to generate instruction-response pairs based the corresponding raw text. the high diversity of the tuning data enables the synthesizer to generalize to unseen data, facilitating the synthesis of instructionresponse pairs for raw pre-training corpora. unlike existing works @xcite using large or closed-source models @xcite to generate synthetic data, we build our instruction synthesizer based on open-source models (typically with 7b parameters), which is much more cost-effective. this efficiency allows us to further scale up task synthesis: augmenting the raw corpora with 200m instruction-response pairs across more than 40 task categories. we conduct experiments in both general pretraining from scratch and domain-adaptive continual pre-training. in pre-training from scratch, our 500m model pre-trained on 100b tokens reaches performance of the 1b model pre-trained on 300b tokens. moreover, models that have undergone instruction pre-training gain significantly more from further instruction tuning. in continual pre-training, instruction pre-training consistently improves performance of llama3-8b foot_1 on two domains: finance and biomedicine, enabling it to be comparable to or even surpass llama3-70b. in summary, our contributions include: • we propose instruction pre-training to explore supervised multitask pre-training, and verify its effectiveness through extensive experiments. ## analysis we conduct a detailed analysis of the instruction synthesizer and the instruction-augmented corpora to understand their impact on lm pre-training. ## related work synthetic instruction generation there have been many works studying synthetic instruction generation, but they mainly focus on posttraining @xcite ), while we focus on pre-training. this makes these works complementary to ours. moreover, our experiments demonstrate that instruction pre-trained models gain more from instruction post-training, highlighting the complementary nature. regardless of the training stage, our method differs from related works in several ways. firstly, we focus on learning from the raw corpora rather than distilling knowledge from strong models @xcite @xcite . secondly, ours can be task-agnostic, in contrast to the more task-specific approaches @xcite @xcite relying on a few gold examples. additionally, we outperforms rule-based methods @xcite data curation for lm pre-training data curation for lm pre-training typically involves collection, cleaning, and organization. most pre-training data are collected from the internet to ensure diversity @xcite @xcite . although diverse, web-scraped data often contain low-quality and duplicate content. therefore, data cleaning techniques are applied to these corpora, including language identification @xcite , perplexity-based @xcite , classifierbased @xcite , and rule-based @xcite filtering. data organization aims at performing more fine-grained programming of the data, including data selection @xcite and constructing training instances related to downstream usage @xcite @xcite @xcite . our work explores an orthogonal direction: augmenting raw corpora with large-scale supervised signals. ## conclusion this paper proposes instruction pre-training to explore supervised multitask learning for pre-training. instead of directly pre-training on raw corpora, instruction pre-training augments the corpora with instruction-response pairs generated by an instruction synthesizer. our instruction synthesizer, finetuned from a highly diverse data collection, is capable of generating diverse instruction-response pairs from various corpora. in pre-training from scratch, instruction pre-training not only outperforms vanilla pre-training on the pre-trained base models but also benefits more from further instruction tuning. in continual pre-training, instruction pre-training substantially enhances the performance of llama3-8b in two different domains. looking ahead, we hope our work can inspire further exploration into this promising area of supervised multitask pre-training, effectively enhancing the general abilities of lms.