Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models (2024)

Abstract: Instruction-tuning language models has become a crucial step in aligning them for general use. Typically, this process involves extensive training on large datasets, incurring high training costs. In this paper, we introduce a novel training data selection based on the learning percentage of the samples. We assert that current language models possess the capability to autonomously select high-quality training data, leading to comparable or improved performance compared to training on the entire dataset. Our experiments span different-sized models, revealing that this characteristic holds for models ranging from 1B (small) to 13B (large) in size. Moreover, we demonstrate an interesting finding that the data hardness transfers across model sizes, and a smaller 350M model can effectively curate high-quality training data with hard samples for a larger 13B model, resulting in an equally or superior instruction-tuned model compared to training on the complete dataset. Utilizing open-sourced OPT and Llama-2 models up to 13B in size, two publicly available instruction-tuning training datasets and evaluated by both automatic metrics & humans, our paper introduces a novel approach to training data selection, showcasing a more efficient alternative.

## introduction instruction tuning empowers large language models (llms) to generalize to novel tasks and instills an instruction-following characteristic, marking the initial stride towards aligning them for general use @xcite @xcite @xcite . this process involves fine-tuning language models with extensive sets of real @xcite and/or synthetic instructions @xcite ., * equal contribution ♠ corresponding author 350m 1.3b 2.7b 6.7b 50 51 52 53 54 51.74 53.21 53.68 53.34 ranking source win rate figure 1: the win rate of opt-13b model trained on 10% data sub-sampled by smaller opt models (350m, 1.3b, 2.7b) from alpaca data, is compared against the opt-13b model trained on the full dataset. all win rates exceed 50, indicating even a smaller 350m dataset can curate high-quality data for a larger 13b model. in this paper, we delve into the measurement of sample difficulty from the model's perspective. drawing inspiration from the learning order metric in @xcite , we propose a novel data selection method that utilizes the learning percentage as a difficulty metric that the model can use to self-rank its training data. essentially, the more learning that occurs in earlier epochs, the easier the sample is considered. we then select the most difficult sample subsets based on this ranking and instruction-tune a language model. our experiments involve two instruction-tuning datasets, alpaca-data @xcite , and dolly @xcite , with performance measured using automated metrics such as al-pacaeval @xcite and human evaluation. our main findings indicate that language models can autonomously select training data, achieving performance equal to or better than training on the entire dataset. furthermore, this characteristic scales across different model sizes, ranging from smaller ones (1b) to larger ones (13b) we publicly release the code here foot_1 . ## experiment setup we design controlled experiments to empirically validate our assertions. our experimental setup encompasses language models spanning various families and sizes, alongside multiple datasets, the specifics of which are detailed below. ## lp learning percentage as a difficulty metric the concept of learning order @xcite ) is designed to assess the quality of a sample in the context of a weaklysupervised classification problem @xcite . the learning order of a data point is defined as the epoch at which it is learned during training, precisely when the model's predicted label aligns with the given ground truth. to adapt this concept to the text a higher learning percentage at earlier epochs indicates that majority of the learning occurs during the initial epochs. given the deep neural models typically learn easier samples initially and progress to more challenging samples later @xcite @xcite , a higher learning percentage in the early epochs implies easy-to-learn samples. since language models are known to learn most of the information in just one epoch @xcite @xcite , we consider lp(1) to rank the training data. the diversity of training data is a pivotal attribute for achieving high quality and optimal performance @xcite . to enhance this diversity, we employ kmeans clustering on the sentence embeddings generated by the all-minilm-l6-v2 model foot_4 on the entire training dataset, ensuring that each cluster contains a minimum average of 50 samples. as a result, the alpaca-data yields 1000 clusters with an average of 52 samples per cluster, while the dolly dataset yields 300 clusters with an average of 50 samples per cluster. subsequently, we rank the training data using lp(1) in ascending order and select the top-k% of samples from each cluster, i.e., the samples that are learned the least in the first epoch. ## lp app a faster & approximate learning percentage metric it is worth noting that to compute lp(1), the model needs to be trained twice-first to obtain the perplexity scores and rank the data, and then to we present the win rates of the model trained on different percentages of selected data using both lp app and lp against the one trained on the complete dataset. to compute lp app (1), we only need to train the model once for 1 epoch, making it more efficient. ## dissecting the difficult data in this section, we analyze the characteristics of samples identified as challenging by the lp metric. we manually examine 250 samples selected from the 1% subset characterized by low lp(1) scores within the alpaca-data corpus, obtained using llama-2 7b. we observe that these difficult samples are longer than the average, maintaining coherence throughout. specifically, the average response length within the 1% low lp(1) subset of alpaca-data is 547 characters, contrasting with the dataset's average of 270. this observation aligns with intuition, suggesting that models encounter difficulty in generating longer and coherent text, thus deeming such instances as challenging. we also found six noisy samples, shown in ## skill-chart analysis in this study, we consider the vicuna-split of the alpaca-eval dataset, which is categorized into nine distinct skill categories, and compare model performances within each skill. the alpaca-data is used as the training dataset for this evaluation. firstly, we compare the performance of the llama-2 13b model trained on 3% of the data selfselected using our proposed lp(1) metric against the same model trained on the complete dataset, as illustrated in figure 8 (a). our findings indicate that performance either improves or remains constant in the math, writing, generic, roleplay, commonsense, and fermi skills when using the 3% self-selected dataset. however, the model trained on the full dataset outperforms in coding, knowledge, and counterfactual skills. this suggests that the data selected using the lp(1) metric is effective and high quality, although certain skills may benefit from a larger data volume. secondly, we examine the performance of a larger model trained on data selected by a smaller model compared to the larger model trained on the full dataset, as shown in figure 8(b) . in this scenario, we use opt-2.7b as the smaller model to select 25% data and opt-13b as the larger model, plotting the win percentage per skill. we notice similar performance trends as previously where the performance either improves or is stable in math, writing, generic, roleplay, and fermi skills with the smaller model's data selection. conversely, the larger model trained on the complete dataset performs better in coding, knowledge, counterfactual, and commonsense skills. this demonstrates that the smaller model can effectively select data for most skills similar to the larger model. finally, we compare the data selected using the alpagasus with data selected using our proposed approximated lp app (1) metric in figure 8(c ). we use the opt-6.7b model and select 10% of the data using both methods. the results show a uniform improvement across all skills with our proposed method, highlighting its superior performance. ## conclusion in this paper, we introduce a learning percentagebased metric for assessing the difficulty of samples. we demonstrate that lms ranging from 1b to 13b sizes can self-select high-quality training data by employing this metric. additionally, we empirically validate the transferability of data hardness across different model sizes, showcasing the efficient curation of high-quality training data by smaller models. furthermore, we propose an optimized version of the metric that offers increased speed while maintaining equal efficacy.