Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks (2022)

Abstract: Teacher-student knowledge distillation is a popular technique for compressing today’s prevailing large language models into manageable sizes that fit low-latency downstream applications. Both the teacher and the choice of transfer set used for distillation are crucial ingredients in creating a high quality student. Yet, the generic corpora used to pretrain the teacher and the corpora associated with the downstream target domain are often significantly different, which raises a natural question: should the student be distilled over the generic corpora, so as to learn from high-quality teacher predictions, or over the downstream task corpora to align with finetuning? Our study investigates this trade-off using Domain Classification (DC) and Intent Classification/Named Entity Recognition (ICNER) as downstream tasks. We distill several multilingual students from a larger multilingual LM with varying proportions of generic and task-specific datasets, and report their performance after finetuning on DC and ICNER. We observe significant improvements across tasks and test sets when only task-specific corpora is used. We also report on how the impact of adding task-specific data to the transfer set correlates with the similarity between generic and task-specific data. Our results clearly indicate that, while distillation from a generic LM benefits downstream tasks, students learn better using target domain data even if it comes at the price of noisier teacher predictions. In other words, target domain data still trumps teacher knowledge.

## introduction in the recent past, large language models (lms; bert-large, @xcite @xcite pretrained in a self-supervised manner on massive web corpora have consistently shown state-of-the-art per- * * corresponding author formance for multiple natural language understanding (nlu) tasks. therefore, it is no surprise that these models are of much interest for virtual assistants such as amazon alexa, apple siri, and google assistant. some studies have shown that these large models trained on generic corpora seem to be more robust to data distributional shifts, relying less on domain-specific training data to perform well @xcite . since large models cannot be directly used for low-latency applications on devices with limited computing capacity, many techniques have been developed to compress them in size. knowledge distillation (referred to simply as distillation hereafter; @xcite , has shown promising results, especially at the high compression rates typically required in nlu @xcite . in this paradigm, lightweight models referred to as students, are trained to mimic the teacher predictions over a transfer set @xcite . when the pretraining and task-specific corpora have significantly different distributions, as is often the case, the choice of data for the transfer set can be ambiguous. on the one hand, using pretraining corpora in the transfer set ensures high quality teacher predictions that are important for effective distillation. on the other, using the downstream corpora, although it might cause noisier teacher predictions, ensures the adaptation of the student to its final use case. to investigate this trade-off, we present a set of experiments where we distill several multilingual students from a large multilingual teacher lm trained using a masked language modeling (mlm) objective. we perform the distillations using transfer sets that comprise of generic and task-specific data in varying proportions. the students are then finetuned and evaluated on two downstream nlu tasks of interest: a domain classification (dc) task and a joint intent classification/named entity recognition (icner) task. for each input utter-ance dc predicts the relevant domain @xcite , ic identifies the user's intent (find a book, play a song, buy an item, etc.) and ner extracts the entities in the utterance (dates, names, locations, etc.). ## relevant work building models with inference speeds that are suitable for production systems is of utmost importance in the industrial setting. therefore techniques for model compression @xcite have been active research topics, with distillation @xcite @xcite showing much promise for nlu models @xcite . distillation processes and their data have evolved over the past few years. in the teacher-student framework proposed by @xcite , they recommend using the original pretraining set as the transfer set. @xcite proposes a more complex two-stage process with generic and task-specific distillation phases, each with their own data sets, designed to augment the performance of the final model towards the task at hand. our work is focused on exploring how varying proportions of generic and task-specific data within the transfer set of a single distillation process impacts downstream nlu performance. since our scope does not include optimizing the distillation process itself, we use a cheaper alternative to @xcite , via a single-stage distillation setup to conduct our exploration (see section a.3 for details). @xcite showed for the pretraining phase, that continued domain-adaptive and taskadaptive pretraining using the downstream task's unlabeled data can improve performance. our work presents similar results for the distillation phase. ## models figure 1 shows a schematic of the models and experimental setup described in this section. ## conclusions we have explored how the use of transfer sets that comprise different ratios of generic to task-specific data, impacts downstream results. encoders distilled from a large teacher perform better than ones trained from scratch, showing that it is better to distill than to directly pretrain, when possible. the largest benefits are shown when using the downstream task's unlabelled data to distill, a student despite noisier teacher predictions. we also find that domains with data that are dissimilar to the generic data show greater performance improvements against a generic baseline when using a stu-dent distilled using task-specific data. these improvements further confirm that distilling using target-domain data can be helpful for downstream performance. finally, we show that if costs permit, teacher-adaptation to the target-domain data via finetuning can result in improved student performance across downstream tasks. chengwei su, rahul gupta, shankar ananthakrishnan, and spyridon matsoukas. 2018. a re-ranker scheme for integrating large scale nlu models. 2018 ieee spoken language technology workshop (slt), pages 670-676. venkat varada, charith peris, yangsook park, and christopher dipersio. 2020. using alternate representations of text for natural language understanding. in proceedings of the 2nd workshop on natural language processing for conversational ai, pages 1-10, online. association for computational linguistics. linting xue, noah constant, adam roberts, mihir kale, rami al-rfou, aditya siddhant, aditya barua, and colin raffel. 2021. mt5: a massively multilingual pre-trained text-to-text transformer. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 483-498, online. association for computational linguistics.