KNN -Instruct: Automatic Instruction Construction with K Nearest Neighbor Deduction (2024)

Abstract: Supervised fine-tuning (SFT) is a critical procedure for aligning large language models. Despite its efficiency, the construction of SFT data often struggles with issues of quality, diversity, and scalability. Many existing methods, inspired by the Self-Instruct framework, typically generate synthetic instructions by prompting aligned proprietary models like ChatGPT. However, such process suffers from stale distribution, resulting in instructions that are merely trivial variations of existing ones. In this paper, we introduce a novel bootstrapping approach termed KNN-Instruct, which incorporates KNN deduction to produce meaningful new instructions by effectively summarizing and learning from similar existing ones. We conduct an economical controlled experiment to preliminarily validate its effectiveness. In the further experiment, we construct a high-quality SFT dataset named KNN-Inst-12k*. Applying the dataset to Qwen-2-7B, we get a MT-Bench score of 7.64, which outperforms all 7B models on the LMSYS leaderboard, including Starling-LM-7B (7.48), OpenChat-3.5 (7.06) and Zephyr-7B-beta (6.53). Our code and data are available at https://github.com/CrossmodalGroup/KNN-Instruct/.

## introduction large language models (llms) pre-trained on large volumes of unlabeled corpus have demonstrated amazing capability in numerous natural language processing (nlp) tasks @xcite . the pre-trained models can be aligned to human intention with the straightforward supervised fine-tuning (sft) @xcite . for the instruction-following task, a sft dataset consists of some conversations (instruction, response). the sft model can be trained with reinforcement learning from human feedback (rlhf) for further alignment (ouyang * corresponding author: zhendong mao. @xcite , which is very intricate and expensive, and has been widely applied in strong models such as chatgpt @xcite . recently, @xcite show that llms could be well aligned to human intention simply through sft, while the scale, quality and diversity of sft dataset are key to the performance of sft model @xcite . to obtain a large, highquality and diverse sft dataset, previous works @xcite usually employ human experts to carefully curate meaningful instructions and corresponding responses, which is very laborious, costly and thus lacks scalability. with the development of instruction-following llms, the past few years have witnessed a lot of works on automatic sft data construction, which employ capable llms like chatgpt instead of human experts to construct meaningful conversations @xcite @xcite @xcite . nevertheless, we notice that current works still more or less fall short of quality, diversity or scalability. for instance, we reproduce self-instruct with the latest gpt-3.5-turbo api, and our case study in appendix b.1 shows that a number of synthetic instructions are not so high-quality, which may be detrimental to the performance of sft model. in this paper, we propose an automatic sft data construction method named knn-instruct. we analyze the types, pros and cons of several existing methods, and mainly make two improvements over the notable work self-instruct: seed pool ranked prompts from crowdsource 2. sample a conv 3. calculate knn for the sampled conv (prompt, response) [exemplar dialog 0] ………… [exemplar dialog k] the passage above shows several dialogs between a human user and the ai assistant. now perform the following task. to validate the effectiveness of our method, we carry out two experiments. in the preliminary experiment, we construct knn-inst-12k, a sft dataset of 12,104 single-turn conversations. we select four competitive methods as baselines, and take necessary measures to ensure fair comparison. we apply these five datasets to two capable pre-trained models, llama-2-7b @xcite and qwen-7b @xcite , respectively. we then evaluate all sft models with alpacaeval @xcite and mt-bench @xcite , which are both widely-recognized llm benchmarks, for five times. the evaluation results show that knn-instruct surpasses all baseline methods on both benchmarks and for both backbones. it is noteworthy that we employ gpt-3.5-turbo for both data construction and llm evaluation in the preliminary experiment to save expenditure. to further explore the performance of knn-instruct, we conduct our second experiment, where gpt-3.5-turbo is replaced with the strong and expensive gpt-4-turbo. we apply the new synthesized dataset, knn-inst-12k* to the recently released pre-trained model qwen2-7b, and evaluate it with mt-bench, where the rigorous gpt-4-turbo serves as a judge. the result is that our qwen2-7b knn-inst-12k* scores 7.64, sur-passing the top three 7b models on the lmsys leaderboard foot_0 : starling-lm-7b @xcite , openchat-3.5 @xcite , and zephyr-7bβ @xcite . specifically, it also outperforms gpt-3.5-turbo and qwen2-7b-instruct in the first turn of mt-bench. in section 5, we design a number of ablation studies to gain a deeper insight into knn-instruct. we thus demonstrate that our improvements on knn deduction and seed dataset do help boost sft performance, and the performance could be further improved by scaling up the dataset or optimize the hyper-parameter k. our contributions are summarized as follows: ## related work automatic sft data construction in recent years, many works have attempted to take advantage of strong instruction-following llms such as chatgpt instead of human experts to construct high-quality, diverse sft data, which remarkably reduce the expenditure and have achieved competitive performance @xcite @xcite @xcite . to some degree, sft data construction is just instruction data construct, because with a instruction, we can easily obtain its response by querying chatgpt. according to different principles, these automatic methods can be roughly divided into three categories: ## knn-instruct in this section, we introduce our proposed knn-instruct in detail. section 3.1 briefly describes the pipeline of knn-instruct. section 3.2, section 3.3 and section 3.4 respectively elaborate on the three key improvements we made over self-instruct. section 3.5 deals with the actual implementation of knn-instruct. ## experiments to justify the validity of knn-instruct, we conduct two experiments: a preliminary experiment in section 4.1, and a further one in section 4.2. ## ablation study we conduct a few ablation studies: on selection of hyper-parameter k, on the procedure of knn deduction, on different seeds, on the scalability of our method, and on the embedding similarity filter, which helps to gain a deeper understanding of knn-instruct. in this section, we use gpt-3.5-turbo-0125 as a judge for mt-bench and alpacaeval, and llama-factory @xcite to evaluate llms on mmlu. ## conclusion in this paper, we introduce an automatic instruction construction framework named knn-instruct. we make improvements on the sampling strategy and seed dataset over current bootstrapping methods. moreover, we remove the redundant instruction elimination procedure to improve efficiency. we conduct a lot of controlled experiments to demonstrate the significance of knn-instruct, where our strongest model qwen2-7b knn-inst-12k* surpasses all current 7b models on mt-bench. we hope this work would contribute to the research of llm alignment.