I ns B ank: Evolving Instruction Subset for Ongoing Alignment (2025)

Abstract: Large language models (LLMs) typically undergo instruction tuning to enhance alignment. Recent studies emphasize that quality and diversity of instruction data are more crucial than quantity, highlighting the need to select diverse, high-quality subsets to reduce training costs. However, how to evolve these selected subsets alongside the development of new instruction data remains insufficiently explored. To achieve LLMs’ ongoing alignment, we introduce Instruction Bank (InsBank), a continuously updated repository that integrates the latest valuable instruction data. We further propose Progressive Instruction Bank Evolution (PIBE), a novel framework designed to evolve InsBank effectively and efficiently over time. PIBE employs a gradual data selection strategy to maintain long-term efficiency, leveraging a representation-based diversity score to capture relationships between data points and retain historical information for comprehensive diversity evaluation. This also allows for flexible combination of diversity and quality scores during data selection and ranking. Extensive experiments demonstrate that PIBE significantly outperforms baselines in InsBank evolution and is able to extract budget-specific subsets, demonstrating its effectiveness and adaptability.

## introduction instruction fine-tuning is widely adopted to refine pre-trained llms to accurately understand human instructions and provide precise, pertinent and harmless responses @xcite . lima @xcite has proved that the quality and diversity of instruction data are significantly more critical than its sheer quantity for training, motivating recent efforts in instruction ## progressive instruction bank evolution in this section, we provide a detailed explanation of pibe, whose pipeline is depicted in figure 2 . ## experiment 4.1 experimental setup candidate instruction data we aggregate five instruction datasets for general instruction following capability: self-instruct @xcite , alpaca (gpt-4) @xcite , dolly @xcite , sharegpt 2 @xcite and wilzardlm (alpaca) @xcite , resulting in a mixed dataset of 278k samples. the statistics of each dataset is presented in ## related work instruction fine-tuning is widely used to refine llms. early methods focused on fine-tuning with large-scale instruction datasets @xcite manually aggregated from extensive nlp task collections @xcite . with advancements in generative models, @xcite has led the trend of synthetic data generation @xcite @xcite . as @xcite found, quality and diversity are more important than quantity, driving recent efforts to cut training costs by removing low-quality and redundant data. existing selection methods can be broadly categorized into three types @xcite : quality-based, diversity-based, and model-specific importance-based selection. quality-based selection humpback @xcite selects high-quality samples through an iterative self-curation process where quality predictions are produced by the fine-tuned model of each turn. recent works typically employ a gpt-model to annotate the data quality. for example, alpa-gasus @xcite employs chatgpt to score the accuracy of instruction data and select data according to a threshold. ## conclusion in this paper, we propose insbank to address the challenge of evolving instruction subset. pibe integrates high-quality and representative data into insbank, striking a balance between data diversity and quality, while maintaining long-term scalability and efficiency. by leveraging a representationbased diversity score with historical information, pibe flexibly combines diversity and quality for data selection and ranking. experimental results show pibe outperforms baselines, providing more optimal and adaptable instruction subsets. the orderliness of insbank also allows users to extract tailored subsets within budget constraints, supporting cost-effective training and the ongoing refinement of llms. this work paves the way for more dynamic and adaptable instruction tuning strategies, enhancing both the efficiency and effectiveness of llm development over time.