Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Tasks (2022)

Abstract: Pre-trained language models (LMs) obtain state-of-the-art performance when adapted to text classification tasks. However, when using such models in real world applications, efficiency considerations are paramount. In this paper, we study how different training procedures that adapt LMs to text classification perform, as we vary model and train set size. More specifically, we compare standard fine-tuning, prompting, and knowledge distillation (KD) when the teacher was trained with either fine-tuning or prompting. Our findings suggest that even though fine-tuning and prompting work well to train large LMs on large train sets, there are more efficient alternatives that can reduce compute or data cost. Interestingly, we find that prompting combined with KD can reduce compute and data cost at the same time.

## introduction state-of-the-art techniques in nlp, such as adapting pre-trained language models (lms) to downstream tasks, typically rely on large model and/or train set sizes @xcite . in real-world applications, serving large models or having large train sets may be prohibitive due to budget constraints, too slow inference, or scarcity of expensive-to-obtain labeled data. thus, solutions to build models that save time, money, and energy are preferable @xcite . a powerful technique to reduce model size is knowledge distillation (kd) @xcite . kd requires the availability of a large unlabeled dataset -the transfer setwhich is usually easy to gather for real-world applications since examples of task inputs are abundant. in kd, the transfer set is weakly labeled with the predictions of a teacher model; then a smaller student model is trained to match these soft labels. the goal is to retain the teacher's abilities in a more compact architecture. in this work, we study how performance varies across different scenarios (i.e., model or train set size) when adapting pretrained lms to downstream tasks. we focus on text classification as a prominent downstream task. we consider the following training procedures to adapt lms for classification: our contributions are two-fold. first, we show that p+kd allows to reduce both model and train set size while retaining high performance. second, we extrapolate recommendations on how to efficiently adapt lms for downstream tasks. the trends we identify indicate that the cost of increasing model parameters or train set size is not always worth it: small models or models trained with little data often achieve comparable performance than larger models trained with more data. ## conclusion and future work we investigated performance-efficiency trade-offs of different training procedures for adapting lms to text classification tasks. we considered efficiency both as the compute cost associated with inference and the data cost of labeling training examples. we found that both prompting and fine-tuning work well to train large lms on large train sets, but there exist more efficient alternatives to build effective models. to reduce compute cost, one can prompt or fine-tune compact lms; if the train set is small, provided the availability of a transfer set, kd from a large model should be applied to obtain a small model. to reduce data cost, prompting is recommended, better if combined with kd for smaller models. to simultaneously reduce both compute and data cost, p+kd is the most efficient training procedure. our results can help nlp practitioners to identify the best strategy to follow on adapting pretrained lms to text classification tasks based on compute budget and the cost of data collection. note that in this work we focused on the compute of cost of inference rather than that of training due to its larger impact on real-world applications where models are served frequently and on a large scale. however, there are differences in training cost among the procedures we investigated (e.g., kd requires training both a teacher and a student model), which will have to be taken into account if facing a limit in train cost budget and resources. for future work, we want to expand our study to other nlp tasks such as sequence tagging, as well as languages other than english. also, in our experiments we used a relatively small teacher model to facilitate experimentation; we expect the performance of p+kd to increase if using larger lms with in-context learning @xcite , as opposed to prompt-based finetuning. teven le scao and alexander m rush. 2021. how many data points is a prompt worth? in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 2627-2636. pengfei liu, weizhe yuan, jinlan fu, zhengbao jiang, hiroaki hayashi, and graham neubig. 2021. pretrain, prompt, and predict: a systematic survey of prompting methods in natural language processing. arxiv preprint arxiv:2107.13586. yu meng, jiaxin huang, yu zhang, and jiawei han. 2022. generating training data with language models: towards zero-shot language understanding. in advances in neural information processing systems. alec radford, karthik narasimhan, tim salimans, and ilya sutskever. 2018. improving language understanding by generative pre-training. timo schick and hinrich schütze. 2021a. exploiting cloze-questions for few-shot text classification and natural language inference. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 255-269. timo schick and hinrich schütze. 2021b. generating datasets with pretrained language models. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 6943-6951, online and punta cana, dominican republic. association for computational linguistics. timo schick and hinrich schütze. 2021c. it's not just size that matters: small language models are also fewshot learners. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 2339-2352. timo schick and hinrich schütze. 2021d. true fewshot learning with prompts-a real-world perspective. arxiv preprint arxiv:2111.13440. emma strubell, ananya ganesh, and andrew mccallum. 2019. energy and policy considerations for deep learning in nlp. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 3645-3650, florence, italy. association for computational linguistics. iulia turc, ming-wei chang, kenton lee, and kristina toutanova. 2019. well-read students learn better: on the importance of pre-training compact models. arxiv preprint arxiv:1908.08962v2. peter west, chandra bhagavatula, jack hessel, jena hwang, liwei jiang, ronan le bras, ximing lu, sean welleck, and yejin choi. 2022. symbolic knowledge distillation: from general language models to commonsense models. in proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 4602-4625, seattle, united states. association for computational linguistics. xingcheng yao, yanan zheng, xiaocong yang, and zhilin yang. 2022. nlp from scratch without largescale pretraining: a simple and efficient framework. in international conference on machine learning, pages 25438-25451. pmlr. jeonghee yi, ye chen, jie li, swaraj sett, and tak w yan. 2013. predictive model performance: offline and online evaluations. in proceedings of the 19th acm sigkdd international conference on knowledge discovery and data mining, pages 1294-1302. xiang zhang, junbo zhao, and yann lecun. 2015. character-level convolutional networks for text classification. advances in neural information processing systems, 28.