How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding (2021)

Abstract: Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge in a large neural network into a smaller one. Even though KD has shown promise on a wide range of Natural Language Processing (NLP) applications, little is understood about how one KD algorithm compares to another and whether these approaches can be complimentary to each other. In this work, we evaluate various KD algorithms on in-domain, out-of-domain and adversarial testing. We propose a framework to assess adversarial robustness of multiple KD algorithms. Moreover, we introduce a new KD algorithm, Combined-KD, which takes advantage of two promising approaches (better training scheme and more efficient data augmentation). Our extensive experimental results show that Combined-KD achieves state-of-the-art results on the GLUE benchmark, out-of-domain generalization, and adversarial robustness compared to competitive methods.

## introduction pre-trained language models have achieved impressive results on a wide variety of nlp problems @xcite @xcite @xcite . the rapidly increasing parameter size, however, has not only made the training process more challenging @xcite , but has also become an obstacle when deploying these models on edge devices. to address the over-parameterization and computation cost of state-of-the-art (sota) pretrained language models, kd @xcite has emerged as a widely used model compression technique in the literature @xcite . recent work on improving kd can be categorized into two directions: 1) designing a better training scheme to help the student model learn efficiently from the teacher model. e.g., matching the student model's intermediate weights and attention matrices with the teacher's during the training @xcite @xcite or designing progressive or curriculum based learning @xcite @xcite to overcome capacity gap @xcite between teacher and student models. the robustness of kd also requires further investigation. recent studies have revealed that the strong performance of neural networks in nlp can be partially attributed to learning spurious statistical patterns in the training set and even the sota models can make mistakes if a few words in their input are replaced @xcite @xcite . as a result, even though kd has achieved good performance in different downstream tasks @xcite @xcite , it is desirable to investigate if these kd methods learn semantic knowledge and are robust enough to retain their performance on an out-of-domain (ood) dataset @xcite or under an adversarial attack @xcite . it would also be desirable to evaluate if different kd algorithms are complimentary and can be combined successfully. our contributions in this paper are as follows: ## methodology first, we evaluate different kd algorithms on three kinds of test sets. in-domain testing where the training and test sets are from the same distribution, ood test where the test set is specially designed to diagnose whether the model overfits to spurious lexical patterns and an adversarial test set to measure robustness to adversarial examples. then we present our comkd algorithm. in figure 1, we present an example from natural language inference where the model can predict an in-domain sample correctly, but makes mistakes on ood (hans) and adversarial (uaf) evaluation. specifically, in this example, the hans sample changes the object and the subject whereas the uaf sample replaces the most semantically important word.