One-Shot Learning as Instruction Data Prospector for Large Language Models (2024)

Abstract: Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality, inadvertently introducing noise that may compromise model performance. To address this challenge, we introduce Nuggets, a novel and efficient methodology that leverages one-shot learning to discern and select high-quality instruction data from extensive datasets. Nuggets assesses the potential of individual instruction examples to act as effective one-shot learning instances, thereby identifying those that can significantly improve performance across diverse tasks. Nuggets utilizes a scoring system based on the impact of candidate examples on the perplexity of a diverse anchor set, facilitating the selection of the most advantageous data for instruction tuning. Through rigorous evaluations on two benchmarks, namely MT-Bench and Alpaca-Eval, our study illustrates that instruction tuning with the top 1% of examples curated by Nuggets substantially outperforms conventional methods employing the entire dataset.

## introduction large language models (llms) @xcite @xcite @xcite have showcased remarkable capabilities @xcite @xcite across a wide range of language tasks by scaling the model size and training data. despite their proficiency, it is imperative to further enhance their alignment with human instructions. this alignment process involves supervised fine-tuning (sft) on input-output pairs, known as instruction tuning. instruction tuning is a crucial step, serving not only to activate the valuable knowledge acquired by llms during pretraining but also to facilitate their interaction with humans in a manner that aligns with natural conversational dynamics. considerable efforts in instruction tuning have been concentrated on collecting larger @xcite , more diverse @xcite @xcite , and intricate @xcite datasets. this is commonly achieved through human crowd-sourcing @xcite @xcite or extracting data from larger pre-existing models @xcite @xcite . despite the growth in the size of datasets employed for instruction tuning, certain studies @xcite @xcite suggest that smaller yet valuable datasets tend to be more effective in harnessing the capabilities of llms. blindly expanding the volume of instruction data without ensuring quality may introduce noise and lead to hallucination issues @xcite . however, there is a lack of standard criteria for selecting high-quality instruction data @xcite @xcite . as depicted in figure 1 , the common practice depends on empirical methods for data selection @xcite , introducing bias in determining data combinations and adjusting based on outcomes. this trial-anderror approach elevates alignment costs for models. we posit that optimal instruction combinations are present within the extensive data available, yet an efficient and cost-effective identification method remains underexplored. in this paper, we introduce nuggets, a simple yet efficient method that harnesses llms as data explorers through one-shot (in-context) learning. this method facilitates extracting high-quality, valuable data from expansive instruction datasets. intuitively, an instructional example holds value in training if it serves as an excellent one-shot demonstration for a specific task. if it can facilitate many tasks, it will be worth being treated as a prime data focus, i.e., "gold instruction". another noteworthy perspective arises from the observation that in-context learning @xcite @xcite employs prompting to implicitly fine-tune the model, while instruction tuning operates through gradient descent. leveraging the performance of in-context learning offers a promising avenue to predict the effects of instruction tuning. concretely, we first select a set that spans multiple tasks, designated as the anchor set, and the dataset of instructions to be optimized is identified as the candidate set. one example is sequentially chosen from the candidate set to act as a one-shot example for in-context learning. subsequently, it is scored based on its impact on the perplexity of each anchor example. this scoring mechanism enables the inference of dependencies between anchor and candidate examples, providing a reference standard for data selection. to evaluate the effectiveness of the proposed nuggets, we conduct extensive evaluations on two widely recognized benchmarks, namely mt-bench @xcite and alpaca-eval @xcite . we choose a popular and powerful llm, llama @xcite , as our base model. experimental findings demonstrate that the nuggets' data filtering strategy engenders a significant improvement in comparison to vanilla fine-tuning approaches. we summarize our main contributions as follows: instruction tuning recent works have introduced a series of techniques that aim to refine large language models (llms), showcasing their ability to generalize effectively to instructions not encountered before. for instance, t5 @xcite pioneered the initial effort of training various natural language processing (nlp) tasks in a unified text-to-text format. flan @xcite introduced the novel concept of instruction tuning, aiming to improve zero-shot task performance by transforming nlp tasks into natural language instructions during model training. furthermore, in-structgpt @xcite handled a wide array of human-created instructions encompassing diverse forms and a broad range of task types tailored for real-world user scenarios. in the absence of the source code release for these notable projects by openai, subsequent efforts by alpaca @xcite and vicuna @xcite were undertaken to explore open-domain instruction tuning, employing the open-source llm llama @xcite . ## nuggets motivation as illustrated in figure 1 , the conventional paradigm for enhancing instructional data in the fine-tuning process of large language models (llms) has predominantly relied on empirical methods. these methods encompass the application of heuristic rules, expert analysis, and iterative adjustments to the data guided by feedback on model performance. notably, this trial-and-error approach imposes significant costs in terms of both human effort and computational resources. recent scholarly consensus suggests that instruction tuning significantly enhances the task generalization capabilities of pre-trained models across various specific tasks @xcite @xcite . in light of this, we posit the hypothesis of a golden instruction: the efficacy of an instructional example is gauged by its influence on the task generalization capability of the model subsequent to the fine-tuning procedure. as the extent of improvement becomes more conspicuous, the instruction gravitates towards classification as "golden instruction". according to this hypothesis, a straightforward method involves fine-tuning an independent model using an instruction example and then comparing the performance of the fine-tuned model with the base model on a predefined dataset containing multiple tasks. this process aims to discern whether the given example qualifies as a "golden instruction". however, this method would lead to an impractical proliferation of fine-tuned models, equivalent to the number of distinct instructions. furthermore, fine-tuning with only a single example may introduce unstable updates to the model's gradients, making it challenging to ascertain the genuine acquisition of the example. motivated by the inherent duality between in-context learning (icl) and gradient descent @xcite @xcite , we "fine-tune" the instruction implicitly through one-shot learning, replacing the need for actually fine-tuning the model. more information can be found in discussion 5. overview the framework of our nuggets is illustrated in figure 2 . firstly, we evaluate the proficiency of llms across a diverse range of tasks using a predefined set of tasks, denoted as the zeroshot score. subsequently, we designate each example from the instruction dataset as a distinct one-shot prompt, concatenating it in front of the predefined tasks. we then recalibrate the model's completion level for these tasks, referred to as the one-shot score. by exploiting the disparity between one-shot and zero-shot scores, we can compute the golden score for each instruction. once the golden scores for all instructions are computed, we can identify the highest-scoring subset, deemed the golden subset, which is subsequently provided directly to the model for the fine-tuning process. ## discussion: one-shot learning as implicit instruction tuning transformer has risen as the prevailing architecture for language models, where self-attention plays a crucial role as a pivotal element within transformer. let x ins , x @xmath0. x ins can be likened to iq k and ia k in equation 3, while x test can be seen as t and ## conclusion this paper presents nuggets, a method leveraging llms to discern more pivotal data for instruction tuning. grounded in one-shot learning, this approach facilitates the identification of examples' value, enabling efficient data selection without dependence on additional annotation and associated costs. benefiting from nuggets, we observe improved instruction following abilities even with smaller training subsets. furthermore, we posit that our method underscores the significance of meticulous data selection, offering valuable insights for future instruction fine-tuning endeavors.