Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models (2023)

Abstract: Fine-tuning large models is highly effective, however, inference can be expensive and produces carbon emissions. Knowledge distillation has been shown to be a practical solution to reduce inference costs, but the distillation process itself requires significant computational resources. Rather than buying or renting GPUs to fine-tune, then distill a large model, an NLP practitioner might instead choose to allocate the available budget to hire annotators and manually label additional fine-tuning data. In this paper, we investigate how to most efficiently use a fixed budget to build a compact model. Through extensive experiments on six diverse tasks, we show that distilling from T5-XXL (11B) to T5-Small (60M) is almost always a cost-efficient strategy compared to annotating more data to directly train a compact model (T5-Small). We further investigate how the optimal budget allocated towards computation varies across scenarios. We will make our code, datasets, annotation cost estimates, and baseline models available as a benchmark to support further work on cost-efficient training of compact models.

## introduction increasing the size of pre-trained models can consistently improve performance on downstream tasks after fine-tuning, as seen in studies based on bert @xcite , roberta @xcite , bart @xcite , t5 @xcite , and the work on empirical scaling laws @xcite @xcite . however, using large models for inference is expensive and contributes to carbon emissions @xcite . to address this, researchers have explored methods to compress large models through techniques such as knowledge distillation @xcite @xcite , which is effective in reducing inference costs @xcite ## study design in this section, we first describe how we formulate the problem for the cost-efficiency study ( §2.1). we then compare two strategies ( §2.2 & 2.3) for building compact models that incur different proportions of computational and human annotation costs. finally, we explain how to estimate the annotation cost ( §2.4) and computational cost ( §2.5) involved in the two strategies. ## evaluating annotation and distillation under a fixed budget in on t5 v1.1 @xcite , which was pretrained in an unsupervised way only, unlike the original t5 @xcite . in the case of fever and natural ques-tions, following @xcite and @xcite ) respectively, we consider a closed-book setting where models should rely solely on its parametric knowledge, and report performances on dev sets as test sets are private. to measure performances, we use accuracy for fever and multipit id , f1 for wlp, stanceosaurus, and natural ques-tions, and bert-ibleu @xcite (i.e., the harmonic mean of self-bleu and bertscore @xcite ) for multipit gen . more details about experimental settings are described in appendix c. ## further analyses in this section, we study varied values of each variable: the initial number (n) of annotated data ( §4.1), the compact model size ( §4.2), and the teacher model size ( §4.3), all of which are fixed in the main experiment ( §3.1). ## gpt-3.as an annotator furthermore, we examine the cost efficiency of gpt-3.5 @xcite annotation through an in-context few-shot learning scheme. @xcite has recently demonstrated that gpt-3 @xcite ) can be used as a cheaper labeler compared to humans. we attempt to scrutinize its applicability to the tasks considered in this work, and also contextualize its result with that of dist. ultimately. we make use of the text-davinci-003 model to generate pseudolabels by prompting with 32 training examples. in this experiment, we assign @xmath10.046 and for stanceosaurus is $0.073, if using gpt-3.5 (details in appendix e). in figure 6 , we compare gpt-3.5 annotation (gpt-3.5 ann.) against the human annotation and distillation strategy. in addition to gpt-3.5 ann., we combine it with human annotation (human + gpt-3.5 ann.) to enhance quality and make a comparison with dist. the results clearly show that while gpt-3.5 could be better than human annotators as hinted in prior work @xcite , it significantly underperforms the distillation (dist.) strategy given the same budget despite gpt-3.5's larger parameters (175b) than the teacher (11b). this once again highlights the different view of knowledge distillation: cost efficiency. ## related work the costs associated with building models have been explored or concerned by many prior works. data annotation. on one hand, researchers have attempted to tackle the problem of noisy or expensive human annotation. for example, @xcite studies how to distribute annotation budgets between more examples with a single label and fewer examples with many labels. chen et al. 2022 ) explores an efficient interactive distillation with meta-learning. the aforementioned works, however, ignore the data budgets and/or barely consider the realistic computational costs involved in the distillation process. while knowledge distillation has been shown effective for compression or generalization in previous nlp works @xcite @xcite , it remains unclear whether or not it is efficient even when considering the actual cost of distillation, which is often overlooked. as concurrent works, @xcite presents a novel principle-driven self-alignment approach, and @xcite introduces a method that involves step-by-step distillation using chain-of-thought @xcite rationales. although the main focus is completely different from ours (i.e., cost), we believe that these works not only enhance this particular area but also have the potential to support our own findings regarding the cost-efficiency of distillation as the new methods would make the gap with annotation even bigger. ## conclusion in this work, we address a dilemma that practitioners often face when building a model: given a limited budget, how to invest it to train a compact model in an economically efficient manner? we provide empirical evidence that (i) only scaling data using human annotators or gpt-3.5 for annotation may not be the most economical solution, and (ii) when adopting the distillation strategy, using a smaller amount of unlabeled data leads to pareto efficient models with a smaller budget, while it becomes more beneficial to use larger amounts of unlabeled data as the budget increases. furthermore, (iii) we demonstrate that in budgetconstrained settings, a smaller final model could produce both better performance and more efficient inference. given these findings, future work can explore different approaches to leveraging a large model's capability such as pruning for cost-efficient compact models.