Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation (2024)

Abstract: With contributions from the open-source community, a vast amount of instruction tuning (IT) data has emerged. Given the significant resource allocation required by training and evaluating models, it is advantageous to have an efficient method for selecting high-quality IT data. However, existing methods for instruction data selection have limitations such as relying on fragile external APIs, being affected by biases in GPT models, or reducing the diversity of the selected instruction dataset. In this paper, we propose an industrial-friendly, expert-aligned and diversity-preserved instruction data selection method: Clustering and Ranking (CaR). CaR consists of two steps. The first step involves ranking instruction pairs using a scoring model that is well aligned with expert preferences (achieving an accuracy of 84.25%). The second step involves preserving dataset diversity through a clustering process. In our experiment, CaR selected a subset containing only 1.96% of Alpaca’s IT data, yet the underlying AlpaCaR model trained on this subset outperforms Alpaca by an average of 32.1% in GPT-4 evaluations. Furthermore, our method utilizes small models (550M parameters) and requires only 11.2% of the monetary cost compared to existing methods, making it easily deployable in industrial scenarios.

## introduction language models (lms) acquire the capability to follow instructions through instruction tuning (it) @xcite @xcite , which aligns large language models (llms) with critical human standards such as security, privacy, and legal compliance. selfinstruct proposes a novel methodology that utilizes lms to construct it datasets @xcite , greatly improving the efficiency of instruction generation. alpaca leveraged a similar strategy @xcite , utilizing text-davinci-003 to construct the alpaca_52k dataset, and subsequent it on llama-7b model @xcite led to the creation of alpaca. despite these advancements, the quality of instructions remains paramount over their quantity. @xcite carefully curated 1,000 instructions, ensuring data quality and diversity by human being, resulting in lima model significantly outperforming the alpaca. nevertheless, creating high-quality instruction sets through manual annotation is both time-consuming and labor-intensive @xcite . a promising approach to mitigate this challenge involves filtering a small subset of high-quality and diverse instructions from the vast amounts of existing instruction data. alpagasus @xcite introduced a straightforward yet effective method that utilizes gpt-3.5-turbo to filter roughly 9k instructions, surpassing alpaca's performance. however, this approach overlooks data diversity, and gpt's evaluations rated 17.3% instruction pairs generated by text-davinci-003 above 4.5 and 74.9% above 4.0, demonstrating gpt's self-enhancement bias @xcite , rendering it unsuitable for assessing instructions generated by models within the same series. therefore, more authentic human preferences should be used to filter instruction sets. moreover, relying on fragile and expensive external gpt apis limits alpagasus in industrial deployment, especially in low-computation resource scenarios. in this work, we propose an effective and efficient method for selecting instruction pairs -clustering and ranking (car). car consists of two steps. the first is ranking through quality estimation on instruction pairs, where an expertaligned scoring model (with 550m parameters only) achieves an accuracy of 84.25% with expert preferences. then, a clustering step ensures the overall diversity of the dataset, minimizing potential capability gaps. our contributions are summarized as follows: in addition, we released our code and models to facilitate future research and industrial endeavors foot_0 . ## experimental setup to compare alpacar with other models, we obtain a single response for each test set sample using a fixed prompt @xcite . judge llms are then compare responses generated by llms ## results and analysis in this section, we compared alpacar with baseline models, including alpaca, alpaca-pandalm, alpaca-cleaned, alpagasus, and vicuna. we replicated all baseline models at a 7b scale and demonstrated the superiority of alpacar at 13b and 30b scales. ## conclusion in this paper, we focus on exploring and resolving the issue of instruction selection during supervised fine-tuning stage. we introduce the car method and examine two perspectives that are warrant considered: (1) evaluating instruction quality using more authentic human preferences: models trained with data annotated by linguistic experts show higher agreement rates and the selected instructions lead to better-performing models. (2) instruction diversity inspires llms' stronger capability: under our selection framework, preserving a small number of instructions for different tasks through cluster improves model performance. experimental results show that fine-tuning llama (ranging from 7b to 30b parameters) with a 1.96% subset of instructions selected by car outperforms models trained on full datasets or data selected by gpt. moreover, data selecting methods using gptfamily or car is a consistent paradigm whether the pre-trained model is more capable or the model parameters scaling up, while those at higher data quality are still challenging. additionally, our approach can be deployed locally without relying on apis, thereby enabling a more efficient instruction selection approach in low-computation resource environments.