From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning (2024)

Abstract: In the realm of Large Language Models (LLMs), the balance between instruction data quality and quantity is a focal point. Recognizing this, we introduce a self-guided methodology for LLMs to autonomously discern and select cherry samples from open-source datasets, effectively minimizing manual curation and potential cost for instruction tuning an LLM. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model’s expected responses and its intrinsic generation capability. Through the application of IFD, cherry samples can be pinpointed, leading to a marked uptick in model training efficiency. Empirical validations on datasets like Alpaca and WizardLM underpin our findings; with a mere 10% of original data input, our strategy showcases improved results. This synthesis of self-guided cherry-picking and the IFD metric signifies a transformative leap in the instruction tuning of LLMs, promising both efficiency and resource-conscious advancements. Codes, data, and models are available.

## introduction large language models (llms) have revolutionized the landscape of artificial intelligence @xcite @xcite . notable models such as gpt-3 @xcite and @xcite leverage extensive datasets and advanced training methodologies to exhibit high-level text understanding and generation capabilities @xcite @xcite @xcite . instruction tuning @xcite is a method employed to refine this work was done when ming li interned at ping an technology (shenzhen) co., ltd., china. in our study, we introduce a novel approach for autonomously identifying the most impactful training samples, which we refer to as "cherry data", from extensive open-source datasets. these data samples are particularly effective in enhancing llm instruction tuning. central to our hypothesis is the idea that llms, through initial training with a small amount of instruction data, can inherently learn to discern and follow instructions, allowing them to estimate the difficulty of instruction data. our method involves a self-guided process that begins with familiarizing the model with a small subset of the dataset during the "learning from brief experience" phase. this phase lays the groundwork for the subsequent "evaluating based on experience" phase, where we introduce the instruction-following difficulty (ifd) score. this metric evaluates how much help the instruction provides to the generation of the corresponding response, by comparing the loss in model responses with and without instructional context. the higher ifd score, indicating less instructional help, suggests a greater difficulty with instructions. on the contrary, the lower ifd score represents that the given instruction can directly benefit the language model largely even without further training, representing the easiness and necessity of the instruction. thus in the final "retraining from self-guided experience" phase, we use data with relatively large ifd scores as the cherry data to train our model, resulting in what we term "cherry models". this methodology, which emphasizes data quality over quantity, differs markedly from existing techniques that rely on external models for data curation. extensive experimental results validate the efficacy of our method. by applying our methodology to the alpaca and wizardlm instruction tuning datasets, our model outperforms the official alpaca model with only approximately 5% data selected and outperforms the reimplemented wizardlm model with approximately 10% data selected. the key contributions of this paper: ## methodology as illustrated in figure 1 , our methodology is divided into three core phases: learning from brief experience, evaluating based on experience, and retraining from self-guided experience. the initial phase emphasizes equipping the model with a basic instruction-following capability. the subsequent phase introduces a novel metric to evaluate the instruction-following difficulty score of each sample based on the previously trained preexperienced model. finally, after obtaining difficulty scores in the target dataset, the cherry samples are selected to train our final model, which we call the cherry models. ## conclusion our study illuminates the potential of harnessing the innate capabilities of llms for selecting high-quality instruction tuning data that fit the model. through our innovative self-guided approach, llms demonstrate the ability to discern and cherry-pick the most pertinent data samples. central to our methodology is the instruction-following difficulty score, a novel metric adept at gauging the nuanced differences between a model's autonomous outputs and expected responses. our findings not only emphasize the importance of data quality over quantity but also underscore the potential for cost-effective llm training.