Knowledge Distillation Meets Few-Shot Learning: An Approach for Few-Shot Intent Classification Within and Across Domains (2022)

Abstract: Large Transformer-based natural language understanding models have achieved state-of-the-art performance in dialogue systems. However, scarce labeled data for training, the large model size, and low inference speed hinder their deployment in low-resource scenarios. Few-shot learning and knowledge distillation techniques have been introduced to reduce the need for labeled data and computational resources, respectively. However, these techniques are incompatible because few-shot learning trains models using few data, whereas, knowledge distillation requires sufficient data to train smaller, yet competitive models that run on limited computational resources. In this paper, we address the problem of distilling generalizable small models under the few-shot setting for the intent classification task. Considering in-domain and cross-domain few-shot learning scenarios, we introduce an approach for distilling small models that generalize to new intent classes and domains using only a handful of labeled examples. We conduct experiments on public intent classification benchmarks, and observe a slight performance gap between small models and large Transformer-based models. Overall, our results in both few-shot scenarios confirm the generalization ability of the small distilled models while having lower computational costs.

## introduction transformer-based language models, such as bert @xcite , contribute widely to the development of dialogue systems. a key component in the development of these systems is natural language understanding (nlu), such as intent classification (ic). intent classification refers to determining the intent of the speaker's utterance in a given domain in dialogue systems. recently, bertbased language models have achieved state-of-theart performance in intent classification through finetuning on task-specific datasets @xcite . however, there are two main challenges in the development of bert-based intent classification models for task-oriented dialogue systems. first, training such models across many domains needs labeled training data from multiple domains. due to the lack of large amounts of multi-domain training data, few-shot learning (fsl) methods, such as metric-based meta-learning techniques @xcite , have been used to adapt bert-based intent classification models to new domains @xcite . in cross-domain few-shot learning methods, the model learns transferable knowledge from large-scale source domain data and generalizes to unseen target domains using only a handful of training samples. the second challenge is the large model size and long inference time of transformer-based models, which hinder the deployment of such models when limited computational resources are available. approaches to reduce the size of models, e.g., knowledge distillation (kd; @xcite , have been introduced. it has been shown that the new compressed models retain a high percentage of the performance while having a shorter inference time than the original models @xcite . task-specific knowledge distillation approaches require sufficiently large training datasets @xcite , ideally with labels @xcite , to distill a powerful small model. however, to obtain both generalized and small models, knowledge distillation methods seem to be incompatible with few-shot learning due to the large need of sufficient training data. therefore, an adaptation of knowledge distillation to few-shot learning is necessary. to the best of our knowledge, task-specific knowledge distillation in cross-domain few-shot learning has largely remained unexplored with a few exceptions in computer vision @xcite and natural language processing (nlp; @xcite . in this paper, we propose a task-specific approach for distilling small models with generaliza-tion ability to new classes and domains in two fewshot learning scenarios: 1) in-domain target class generalization in single-and multi-domain intent classification; 2) target domain adaptation in multidomain intent classification. to this end, we first pretrain a transformer-based prototypical teacher network @xcite on source classes and domains using meta-learning. then, we design a prototypical student network and pass the transferable knowledge to the student using knowledge distillation. during the distillation process, we consider a prototype loss as a new component in the standard distillation loss function. this loss measures how much each prototype that is produced by the student model resembles the respective prototype produced by the teacher model. moreover, as opposed to standard batch training in knowledge distillation, we introduce an episodic distillation process. this way, we obtain a small student model that is compatible with few-shot scenarios and generalizes to unseen target classes and domains. our contributions are summarized as follows: 1) we propose a new knowledge distillation approach compatible with few-shot learning by introducing an episodic distillation process and using the prototype-based distillation loss. our novel approach combines advantages of few-shot learning with knowledge distillation. 2) we perform extensive experiments on four public nlu benchmarks and compare the distilled small model with the large model in the few-shot intent classification scenario. results show a slight performance drop for the small model while having lower memory consumption and a slightly faster inference speed. 3) we show that the small model can effectively generalize and adapt to target domains without the teacher supervision in the few-shot target domain adaptation. this is a more challenging and realistic scenario for small student models. ## approach we first describe the teacher and student model architectures, followed by our proposed model training procedure. we elaborate on details of the proposed episodic distillation process and show how our approach preserves the generalization ability of the distilled models under few-shot learning scenarios. ## experiments we conduct extensive experiments to evaluate the proposed approach on public intent classification datasets. we simulate two scenarios: in-domain target class generalization and the more challenging scenario, target domain adaptation in multi-domain intent classification. experiments have been implemented in pytorch and performed on a single nvidia 8gb gpu in ubuntu 16.04.6 lts. ## conclusion we address the nontrivial merging problem of metalearning and knowledge distillation. our proposed approach distills large transformer-based models into smaller student models, which are compatible with few-shot learning scenarios in intent classification. through a multi-step meta-training with an episodic knowledge distillation, we obtain a small distilled model that is generalizable and adaptable to new classes and domains using only a few labeled examples. our results in target domain adaptation show that the small model can adapt effectively to new domains without teacher supervision. this removes the need for a large teacher when time and computational resources are limited. compared to the large model, we observe a slight performance loss and less memory consumption in the distilled model. in summary, our results provide insights into the advantages and limitations of a joint few-shot learning and knowledge distillation approach to foster future research in this area. our primary findings suggest that it is worthwhile to explore different fsl techniques jointly with kd for cross-domain few-shot performance improvements. overall, this topic still merits more attention to aid the practical deployment of nlu models in dialogue systems under low-resource scenarios. as future research, we will study novel joint methods for the cross-domain generalization problem under low-resource scenarios. moreover, we will investigate the methods in joint nlu tasks, specifically slot filling and ic.