SMASH : Improving SMA ll Language Models’ Few- SH ot Ability with Prompt-Based Distillation (2022)

Abstract: Large-scale language models coupled with prompts have shown remarkable performance on few-shot learning. However, through systematic experiments, we find that the few-shot performance of small language models is poor, and using prompts on them brings fewer improvements than on larger ones. In this paper, we propose SMASH, an approach to improve SMAll language models’ few-SHot ability by training on intermediate tasks before prompt-based fine-tuning on downstream tasks. We design intermediate tasks for sentence-pair tasks and sentiment classification tasks by creating training examples with prompt templates similar to downstream tasks using sentences sampled from a large-scale unsupervised corpus, and apply knowledge distillation to distill from outputs of larger pre-trained models as the training objective. We conduct extensive experiments and show that SMASH can make a 6-layer DistilRoBRETa-base achieve comparable performance on few-shot datasets with a 12-layer RoBERTa-base at a low cost.

## introduction language models at scale, such as @xcite , have shown remarkable performance on prompt-based few-shot learning on a wide variety of tasks given only a natural language prompt and few demonstrations. however, the ability of few-shot learning usually comes with heavy computation and a huge amount of parameters. recent works @xcite investigated prompt-based few-shot learning on moderately-sized language models such as bertlarge @xcite , roberta-large @xcite and gpt-2 @xcite , but in this paper, we investigate whether we can make small language models, such as distilroberta-base @xcite , better fewshot learners. prompt-based fine-tuning has been seen as a promising method for few-shot learning as it uses language modeling heads instead of introducing new parameters as task-specific classification heads during fine-tuning, thus narrowing down the gap between pre-training (e.g., masked language modeling for roberta) and applying to downstream tasks. however, to tackle this problem, we propose smash, an approach of further training small language models on intermediate tasks before applying them to few-shot downstream tasks. intuitively, if we can design intermediate tasks similar to both the pre-training task and downstream tasks, we can mitigate this problem by replacing one large gap (from the pre-training task to downstream tasks) with two smaller gaps (from the pre-training task to intermediate tasks, then to downstream tasks). noticing that same manual prompt templates (or similar templates with minor differences) are often used when prompt-based fine-tuning models on a group of similar tasks (e.g., @xmath0, x 1 .</@xmath1, such as mnli, qnli, rte, etc.), we consider using this prompt template and sample sentences from a large-scale unsupervised corpus to form the inputs of the intermediate task. to construct supervision signals we leverage knowledge distillation @xcite by feeding the inputs to a larger pre-trained language model and using its outputs as the training objective. in this way, the intermediate task can be both similar to the pre-training task (by training on similar distributions of data, e.g. large scale corpus from the web) and downstream tasks (by using similar prompt templates). from the perspective of knowledge distillation, the intermediate task can also be seen as a kind of data augmentation using a largescale unsupervised corpus to transfer knowledge of solving a group of similar tasks from larger models to smaller models, which can be exploited later by prompt-based fine-tuning on downstream tasks. as the intermediate task depends on the input format of downstream tasks, it's not feasible to experiment with smash on all nlp tasks at once. in this paper, we only take sentence-pair tasks and sentiment classification tasks, two groups of tasks that are popular in nlp as an example, and design two intermediate tasks respectively. note that practitioners can also use smash on other groups of downstream tasks by designing their own intermediate tasks under the training framework we proposed. experiments on the glue benchmark and several other tasks show that using smash can make a 6-layer distilroberta-base achieve com-parable performance with a 12-layer robertabase on few-shot datasets at a low cost. we find that smash provides more improvements on more complicated tasks like natural language inference and sentence similarity than easier tasks like sentiment classification, and is robust over different templates, verbalizers, and model structures. in summary, our key contributions are: ## conclusion in this paper, we present smash, an approach of using knowledge distillation on an unsupervised corpus to improve small language models' few-shot performance. the principle of this approach is distilling the model using input similar to downstream tasks sampled from unsupervised corpus as an intermediate task to transfer knowledge of solving these tasks and further narrow down the gap between pre-training and prompt-based fine-tuning. we design intermediate tasks for sentence-pair tasks and sentiment classification tasks. we show that our approach results in significant improvements on few-shot datasets, especially for harder tasks like natural language inference. we analyse smash on different distillation objectives, and verify its robustness over different templates, verbalizers, and model structures. possible future directions of this work include: apply smash on more types of downstream tasks, especially those that can not be easily formulated using prompts or are difficult to simulate using unsupervised corpus (e.g., text-to-sql); or explore intermediate tasks that are more data-efficient.