Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning (2024)

Abstract: Instruction tuning is critical to large language models (LLMs) for achieving better instruction following and task adaptation capabilities but its success heavily relies on the training data quality. Many recent methods focus on improving the data quality but often overlook the compatibility of the data with the student model being finetuned. This paper introduces Selective Reflection-Tuning, a novel paradigm that synergizes a teacher LLM’s reflection and introspection for improving existing data quality with the data selection capability of the student LLM, to automatically refine existing instruction-tuning data. This teacher-student collaboration produces high-quality and student-compatible instruction-response pairs, resulting in sample-efficient instruction tuning and LLMs of superior performance. Selective Reflection-Tuning is a data augmentation and synthesis that generally improves LLM finetuning and self-improvement without collecting brand-new data. We apply our method to Alpaca and WizardLM data and achieve much stronger and top-tier 7B and 13B LLMs.

## introduction the quality of instruction tuning @xcite @xcite @xcite data is paramount to the llm being fine-tuned, i.e., the student model. there is a growing trend and demand for the community to automatically improve the quality of instruction tuning data. previous works either curate datasets by human experts @xcite @xcite or distill the responses of well-trained llms @xcite @xcite @xcite @xcite . the selfimprovement @xcite @xcite ability of llms has also been explored to improve the instruction or response of a training sample. however, these existing methods of data enhancement @xcite @xcite do not take a critical criterion into account: is the teacherrefined data compatible to the needs of the student model? these approaches typically do not account for the inherent randomness and potential degradation associated with the generative models' output, leading to an oversight in how the student model responds to these "improved" data samples. thus a mechanism for the student model to selectively integrate these enhancements has been notably absent. to bridge this gap, our work introduces an teacher-student collaboration pipeline wherein a teacher generative model engages in a reflection process to enhance both the instruction and response of a data sample. the student model then evaluates whether to incorporate these improvements based on its unique statistical attributes. this pipeline is versatile and can be adapted to various contexts where data enhancement is needed. then, another pivotal question arises: how does the student model decide which enhanced data are needed and critical to its training? this question underpins the challenge of autonomously evaluating the quality of instructions and responses. common practices involve utilizing sophisticated models like gpt-4 for assessment purposes @xcite @xcite or employing a secondary judge model equipped with evaluative capabilities @xcite . these methods, however, present limitations: they fail to address the discrepancies between the evaluating model and the actual student model undergoing training. particularly in the latter approach, even though the judge model and the student model might share the same structural framework, their weight distributions diverge once endowed with the evaluative functions. consequently, the preferences of the judge model may not align with the real student model's requirements. to circumvent these issues, we adopt a statistical method, utilizing the instruction-following difficulty (ifd) score proposed by @xcite . this score is derived directly from the raw student model, thereby mitigating potential domain shifts and ensuring that the evaluation is better aligned with the student model's learning context. in our approach, the ifd score serves as a crucial metric that measures how much help the instruction can provide to the likelihood of the response if added as an extra condition, representing the difficulty of the sample. however, though effective, the ifd score mainly assesses the instructions. motivated by humpback @xcite which requires llms to generate potential instruction based on responses, we further introduce a reversed version of ifd named reversed-ifd (r-ifd). this metric evaluates how much the response contributes to predicting the corresponding instruction. a lower r-ifd score suggests the student can easily deduce the corresponding instruction given the response, indicating this sample is feasible for the student to learn, representing the feasibility of the sample. this dual approach, employing both ifd scores for difficulty and r-ifd scores for feasibility, enables a comprehensive and nuanced assessment of the instruction-tuning process, ensuring the refined data aligns well with the student model's capabilities and objectives. we name our overall method selective reflection-tuning, which contains the selective instruction reflection phase and the selective response reflection phase. in the first phase, a teacher model is utilized to reflect on the instruction of the given sample based on some criteria and generate a new sample. then the student model makes the decision of whether to accept the improvement based on difficulty (ifd). in the second phase, the teacher model reflects and generates a sample with a new response and the student model decides whether to accept based on feasibility (r-ifd). with our interactive pipeline, we obtain a dataset with supreme quality, with only instruction tuning on a relatively small amount of data, our model outperforms most existing open-source models with even larger model sizes. our contributions include: ## preliminaries let f θ denote the pre-trained student model, e.g., llama, with parameters θ and g the teacher model, e.g., chatgpt. let lowercase letters x, y, z, c, .. denote the text segments, which could be phrases or sentences, and each token in x is denoted as x[i]. we use uppercase letters d, .. to denote the collection of language sequences or datasets, and d 0 represents the initial base dataset. since both f θ and g are in auto-regressive manners, a sequence @xmath0, ..., x[n]) can be further denoted as: in the instruction tuning setting, there will be a mapping function that turns the original raw instruction x into the desirable format and requests models for a response y. for simplicity, we directly notate this process as y ∼ f (y|x). and the loss function for instruction-tuning can be denoted as: ## methodology as shown in figure 1 , there are two main phases in our method, selective instruction reflection and selective response reflection phase. in each phase, the teacher model generates the updated version of instructions or responses based on some given specific criteria {c ins,1 , ..., c ins,k } 1 , then the student model judges if the updates are beneficial to it based on difficulty (ifd) or feasibility (reverse-ifd). finally, these selectively improved samples can be used for the final instruction tuning. ## comparison with related work earlier works on instruction tuning focus on creating large, high-quality datasets curated by human experts @xcite @xcite @xcite , time-consuming and labor-intensive. thus a number of works try to construct instruction-tuning datasets automatically. self-instruct @xcite utilizes the in-context learning capability of gpt-3 to expand tasks to many diverse instructionresponse pairs. wizardlm @xcite applies an evolution methodology to refine and diversify the original instruction data. lamini-lm @xcite introduces to generate top-fuided instructions based on wiki data. @xcite utilize gpt4 to generate responses for existing datasets. ultrachat @xcite , establishes various scopes and systematically generates a multitude of instructions within each designated area. orca @xcite directly apply gpt4 to generate reasoning steps for given instructions. selfee @xcite utilizes chatgpt to enhance the response quality. reflection-tuning @xcite improves both the instruction and response sequentially by reflecting on specific criteria. deita @xcite utilizes chatgpt to diversify and then select the data. lift @xcite also tries to utilize chatgpt/gpt4 to expand and compress the data. all the above works are related to ours by involving a teacher model to improve the instruction data, however, all of them are teacher-dominating: both the generation and selection are all decided by the teacher model and without involving the student. we are the first to introduce the teacherstudent collaboration pipeline and it works fine. ## conclusion selective reflection-tuning, as proposed in this paper, marks a significant advancement in data improvement for instruction tuning of large language models. by integrating an interactive pipeline between a teacher model and a student model, and utilizing the novel metrics of ifd and reversed-ifd, this approach has demonstrated a marked improvement in the quality and relevance of instruction-tuning datasets. the resulting enhancement in model performance across various benchmarks not only attests to the efficacy of our method but also suggests its potential applicability in broader machine learning contexts.