Adaptive Contrastive Knowledge Distillation for BERT Compression (2023)

Abstract: In this paper, we propose a new knowledge distillation approach called adaptive contrastive knowledge distillation (ACKD) for BERT compression. Different from existing knowledge distillation methods for BERT that implicitly learn discriminative student features by mimicking the teacher features, we first introduce a novel contrastive distillation loss (CDL) based on hidden state features in BERT as the explicit supervision to learn discriminative student features. We further observe sentences with similar features may have completely different meanings, which makes them hard to distinguish. Existing methods do not pay sufficient attention to these hard samples with less discriminative features. Therefore, we propose a new strategy called sample adaptive reweighting (SAR) to adaptively pay more attention to these hard samples and strengthen their discrimination abilities. We incorporate our SAR strategy into our CDL and form the adaptive contrastive distillation loss, based on which we construct our ACKD framework. Comprehensive experiments on multiple natural language processing tasks demonstrate the effectiveness of our ACKD framework.

## introduction recently, deep learning @xcite @xcite has achieved success in many natural language processing tasks. however, due to limited computation and storage resources, current deep learning approaches are hard to be deployed on mobile devices. knowledge distillation is an effective approach to compress the model for mobile deployment, which aims to use a pretrained teacher network to help the training of a lightweight student network. to achieve this, the student needs to learn discriminative features. namely, we need to push the features * equal contribution. † corresponding author. teacher features student features i have dogs. (a) how are you? teacher features (c) teacher features student features (b) student features optimization direction of traditional knowledge distillation methods optimization direction of our cdl/a-cdl optimization direction of our ackd framework sample from class 1 sample from class 2 i have dogs. i don't have dogs i have dogs i don't have dogs of the sample from different classes (negative pairs) far away from each other and keep the features of the samples from the same classes (positive pairs) close. current knowledge distillation methods for bert implicitly learn discriminative student features. they assume the teacher is well-learned (i.e., features of negative pairs are far away from each other in the teacher). then, they minimize the feature distance of each sample between the teacher and student to make the student feature discriminative, as shown in fig. 1(a) . in this way, the features of negative pairs in the student can be pulled far away from each other. however, the aforementioned assumption is not always held. commonly used words will appear in the sentences with different meanings, causing the features of negative pairs in the teacher to be close to each other, as shown in fig. 1(b) . in this case, training the student using the current knowledge distillation paradigm will result in the features of negative pairs in the student being close to each other as well. so, it is desirable to in- ## adaptive contrastive knowledge distillation in this section, we will introduce our adaptive contrastive distillation (ackd) framework. the goal of our ackd framework is to use a pre-trained teacher model with a large capacity to help the training of a lightweight student model, and its overview is shown in fig. 2 . the loss of our ackd framework when training the student comes from four parts: cross-entropy loss (cel), knowledge distillation loss (kdl), patient loss (ptl), and our adaptive contrastive distillation loss (a-cdl). ## experiments in this section, we perform comprehensive experiments and extensive ablation studies. ## conclusion in this paper, we have proposed a new knowledge distillation approach called adaptive contrastive knowledge distillation (ackd) for bert compression. we first introduce a novel contrastive distillation loss (cdl) as the explicit supervision to learn more discriminative student features. then, we propose a new strategy called sample adaptive reweighting (sar) to adaptively pay more attention to hard samples with fewer discrimination abilities. the sar strategy can be seamlessly incorporated into the cdl and form the adaptive contrastive distillation loss (a-cdl). based on a-cdl, we construct our ackd framework, where dynamic feature storage is used for better sample diversity. extensive experiments on multiple natural language processing tasks demonstrate the effectiveness of our ackd framework for bert compression.