An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models (2024)

Abstract: Supervised finetuning (SFT) on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities observed in modern large language models (LLMs). However, the annotation efforts required to produce high quality responses for instructions are becoming prohibitively expensive, especially as the number of tasks spanned by instruction datasets continues to increase. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool, but its high computational cost remains a barrier to its widespread applicability in the context of LLMs. To mitigate the annotation cost of SFT and circumvent the computational bottlenecks of active learning, we propose using experimental design. Experimental design techniques select the most informative samples to label, and typically maximize some notion of uncertainty and/or diversity. In our work, we implement a framework that evaluates several existing and novel experimental design techniques and find that these methods consistently yield significant gains in label efficiency with little computational overhead. On generative tasks, to reach the same generalization performance, our methods save 50% of the annotation cost compared to random sampling.

## introduction supervised finetuning (sft) on instruction datasets has shown immense potential in improving the zero-shot performance of large language models (llms) @xcite . recent developments in this field have been fostered by the availability of large-scale instruction datasets, consisting of natural language instructions with desired responses based on human judgment @xcite @xcite . throughout the community, there have been several efforts to further increase the number of tasks included in these * equal contribution, alphabetically ordered. datasets to improve llm generalization @xcite @xcite . in addition, supervised finetuning is especially important for handling novel forms of tasks. for example, to develop new multimodal capabilities or defend against novel prompt hacking strategies, one must need human generated responses to finetune existing state-of-art models like gpt-4. a crucial bottleneck of sft is the need for annotating a massive set of instructions with detailed responses, which can be done either manually or automatically. for improving state-of-art models, one necessarily need to rely on manual approaches. this involves the use of crowd-workers or expert annotators, who produce high quality responses for almost any task but can become extremely expensive at scale due to the labor intensive process of annotation. to improve small scale and more domain-specific language models, automatic annotation methods have been proposed to reduce the burden on human annotators by labeling instructions using preexisting llms @xcite @xcite . however, models like gpt-4 are still costly to query, making the annotation for a large set of prompts potentially prohibitive. in addition, for specialized domains, human experts are still crucial in this annotation process as general llms may not provide high quality responses. therefore, we seek to answer the following question: can we propose label-efficient strategies that annotate fewer prompts while obtaining equally good generalization performance? towards improving the label efficiency of deep models, researchers have been predominantly focusing on active learning @xcite @xcite @xcite . these techniques have been proven useful in some relatively easier nlp tasks such as machine translation @xcite figure 1 : comparison between different annotation schemes for label-efficient sft. random sampling simply chooses prompts uniformly at random which underperforms as it is prone to redundancy and may oversample from the major modes. on the other hand, one can choose them more strategically both through active learning and experimental design. active learning, however, is an adaptive procedure and requires computationally expensive model retraining and inference for every batch of annotation. in this paper, we study the problem through the lens of experimental design, which enjoys increased label-efficiency compared to random sampling, while incurring minimal computation cost compared to active learning. and classification @xcite , but remain under-explored in general natural language generation tasks. as shown in figure 1 , the active learning framework incrementally adds batches of samples to the labeled pool by repeatedly: (1) training a model on the currently labeled data and (2) using some model dependent measure of informativeness to select a new batch of points to query the annotator @xcite . however, each iteration of active learning requires performing inference on all unlabeled samples and retraining the model on the expanded set of labeled samples @xcite . in the context of parameter-heavy llms where inference and training are expensive, the computational cost associated with active learning may outweigh its potential savings in annotation costs. in this work, we propose leveraging experimental design to select the optimal set of instructions to annotate. experimental design concerns situations where we have to organize an experiment in order to gain some information about an object of interest. in the context of training an ai system, the "experiment" involves selecting a subset of unlabeled examples from a large pool for labeling in order to create a training set to learn a good model (the object of interest, in this case). as demonstrated in figure 1 , in contrast to active learning where the labeled set is expanded incrementally, experimental design techniques select the set of samples to label in a single step based solely on the initial model. this circumvents almost all of the computational cost of active learning, allowing for gains in label efficiency to be realized with practically no overhead. while experimental design has been studied theoretically, its empirical benefits have been underexplored. we introduce a framework for evaluating experimental design techniques for finetuning llms, and propose a suite of experimental design techniques that demonstrably improve the label efficiency of sft. we develop novel scores, such as maximum token uncertainty, that quantify the llms uncertainty on a particular sample and correlate well with its usefulness as training data. we also propose a class of experimental design heuristics that employ the facility location function as an objective, to select a diverse and representative set of samples for annotation. overall, our contributions can be summarized as follows: (1) we are the first, to the best of our knowledge, to utilize experimental design for sft (2) we introduce a framework to perform comprehensive evaluation on existing experimental design techniques (3) we propose a suite of novel strategies that improve the label efficiency, significantly outperforming random sampling by more than 2% accuracy across different annotation budgets (4) compared to previous works @xcite , our work is the first to see an-notation cost savings on generative tasks. to reach the same generalization performance, our methods save 50% of the annotation cost compared to random sampling (unlike random sampling that fails to achieve the same generalization). our study focuses on experimental design, rather than active learning. active learning generally requires retraining the model and forward inference on the entire pool after each batch of labels is gathered in order to select the next batch for labeling. the retraining and repeated inference can be computationally expensive, particularly in the case of llms. experimental design does not require additional computation of this sort, since the subset selection of unlabeled examples is done just once before any labels are collected. deep active learning data selection strategies for label-efficient learning have been largely studied under the framework of deep active learning, which sequentially and adaptively choose informative examples to annotate. deep active learning methods typically use measures of uncertainty @xcite @xcite @xcite , diversity @xcite @xcite , or some combination of both @xcite @xcite @xcite in order to determine a set of useful samples to annotate. in addition to classical instancelevel selection, there are also many existing works focused on task-wise selection @xcite @xcite . while we propose an experimental design framework for the label-efficient sft problem, our strategies are inspired by the latest instance-level deep active learning literature and adapted to work under this framework for supervised finetuning of llms. a few works have considered the application of active learning for sft or other closely related settings. @xcite explores the use of active learning to improve the label efficiency of the closely related task of natural language generation. however, this work reports inconsistent findings showing that there are limited settings where active learning demonstrates any significant advantages over random sampling. @xcite is the most similar to our work, proposing an active learning framework for instruction tuning. unlike our work, @xcite performs task-level selection, as opposed to instance-level selection. in other words, their framework estimates the usefulness of each unlabeled task and annotates all instances within the tasks that are deemed most useful. however, task-level selection makes the simplifying assumption that every instance is equally useful within a task, which may inhibit the quality and reduce the resolution of the selected subset. moreover, the approach in @xcite suffers from the aforementioned computational cost of active learning which is avoided in our framework with the use of experimental design. data selection for sft many recent works demonstrate that small subsets of instruction data can be sufficient for finetuning an llm. @xcite demonstrate that finetuning an llm on a dataset consisting of only manually-curated 1000 instruction/response pairs is sufficient to achieve strong generalization, but do not propose any general algorithmic procedure for subset selection. other works propose instruction dataset pruning techniques that select subsets based on some combination of quality, diversity, and/or difficulty @xcite @xcite @xcite . these approaches use both the instruction and its corresponding response to choose which training samples should be retained to improve the computational efficiency of sft. unlike these approaches, our framework is designed to maximize label efficiency and assumes that the response to an instruction are not available until selected for annotation. finally, @xcite propose techniques to reduce the annotation cost to construct validation sets that are used for model selection, which is complementary to the method proposed in our work. ## prompt selection strategy under the experimental design framework, the learner is given a set of initial n prompts @xmath0, x 2 , ..., x n }, where each prompt x @xmath1, x @xmath2,1 , ..., x i,ℓ i }, where x denotes the domain of all possible sequence of input tokens. additionally, we let g denote the pretrained language model. given a budget of @xmath3, a selection strategy chooses k prompts from x, denoted as s ⊂ x, based on different measures of informativeness of annotating an example to the pretrained model g. one then gathers well-written responses to prompts in s from annotators (e.g., human experts or advanced llms). experimental design aims to optimize the performance of the model g ′ finetuned on the selected prompts s and their responses. ## experiments in this section, we compare multiple experimental design strategies against random sampling and observe significant improvements. we describe the various setups in section 4.1 and report the evaluation results based on common benchmarks (section 4.2) and comparisons by gpt-4 (section 4.3). lastly, we conduct ablation study and document the selection of hyperparameters in section 4.4. details about our experimental compute cost and complexity can be found in appendix 5. ## computational complexity we primarily used a100 and a40 gpus. we precompute the embeddings which takes 12 hours. for a100, each trial takes roughly 24 gpu hours including the evaluation for 90k budget, 15 gpu hours for 45k and 9 hours for 30k respectively. for a40, roughly 30 gpu hours for 90k budget. ## conclusion and future work overall, we proposed using experimental design for finetuning llms with fewer annotated instructions. we present several heuristics that can be used to determine which set of instructions to annotate, and for the first time, demonstrate significant gains in label efficiency on generative tasks. future research directions may include the following: • our work demonstrates the first empirical results of experimental design for sft, but future work can focus on devising new methods within this framework to further improve label efficiency.