REI nstruct: Building Instruction Data from Unlabeled Corpus (2024)

Abstract: Manually annotating instruction data for large language models is difficult, costly, and hard to scale. Meanwhile, current automatic annotation methods typically rely on distilling synthetic data from proprietary LLMs, which not only limits the upper bound of the quality of the instruction data but also raises potential copyright issues. In this paper, we propose REInstruct, a simple and scalable method to automatically build instruction data from an unlabeled corpus without heavy reliance on proprietary LLMs and human annotation.Specifically, REInstruct first selects a subset of unlabeled texts that potentially contain well-structured helpful and insightful content and then generates instructions for these texts. To generate accurate and relevant responses for effective and robust training, REInstruct further proposes a rewriting-based approach to improve the quality of the generated instruction data. By training Llama-7b on a combination of 3k seed data and 32k synthetic data from REInstruct, fine-tuned model achieves a 65.41% win rate on AlpacaEval leaderboard against text-davinci-003, outperforming other open-source, non-distilled instruction data construction methods. The code is publicly available at https://github.com/cs32963/REInstruct .

## introduction instruction tuning @xcite is an important step in aligning large language models (llms) @xcite . by tuning llms @xmath0, @xmath1, instruction tuning enhances zero-shot performance of llms across a variety of tasks @xcite , and paves the way for further preference training such as reinforcement learning from human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite . manual annotation distill from proprietary models build from unlabeled corpus annotators annotated data proprietary model seed data synthetic data unlabeled text synthetic data difficult and costly, hard to scale restricted usage, bounded quality diverse, high-quality, and scalable ## method the goal of our method is to automatically build instruction data from an unlabeled text corpus. in this section, we first outline how we select unlabeled texts that contain well-structured helpful and insightful contents ( §3.1). next, we describe how to convert these contents @xmath2, @xmath3.2) and response rewriting ( §3.3). finally, we explain how to instruction finetune pretrained llms using both seed instruction data and our synthetic data ( §3.4). figure 2 illustrates the overall framework of our method. ## conclusion in this paper, we propose reinstruct, a simple and scalable method to automatically build instruction data from unlabeled corpus. our method only requires small amount of labeled data, and does not heavily rely on proprietary llms. experiment results show that instruction tuning llms on a small set of human-labeled data and our synthetic data achieves competitive instruction following performance, demonstrating the quality of our synthetic data. we believe this work provides insights for instruction data annotation and helps foster research in instruction tuning. for future work, we aim to develop more sophisticated neural methods to collect and extract instruction following signals from unlabeled dataset across different modalities.