Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021)

Abstract: Recent studies on compression of pretrained language models (e.g., BERT) usually use preserved accuracy as the metric for evaluation. In this paper, we propose two new metrics, label loyalty and probability loyalty that measure how closely a compressed model (i.e., student) mimics the original model (i.e., teacher). We also explore the effect of compression with regard to robustness under adversarial attacks. We benchmark quantization, pruning, knowledge distillation and progressive module replacing with loyalty and robustness. By combining multiple compression techniques, we provide a practical strategy to achieve better accuracy, loyalty and robustness.

## introduction recently, many large pretrained language models (plms, @xcite @xcite @xcite have been proposed for a variety of natural language processing (nlp) tasks. however, as pointed out in recent studies @xcite @xcite , these models suffer from computational inefficiency and high ecological cost. many attempts have been made to address this problem, including quantization @xcite , pruning @xcite , knowledge distillation (kd) @xcite @xcite @xcite @xcite and progressive module replacing @xcite . bert @xcite is a representative plm. many works compressing bert use preserved accuracy with computational complexity (e.g., speed-up ratio, flops, number of parameters) as metrics to evaluate compression. this ## bert compression compressing and accelerating pretrained language models like bert has been an active field of research. some initial work employs conventional methods for neural network compression to compress bert. for example, q8-bert @xcite and q-bert @xcite employ weight quantization to reduce the number of bits used to represent a parameter in a bert model. pruning methods like head prune @xcite and movement pruning @xcite remove weights based on their importance to reduce the memory footprint of pretrained models. another line of research focuses on exploiting the knowledge encoded in a large pretrained model to improve the training of more compact models. for instance, distilbert @xcite and bert-pkd @xcite employ knowledge distillation @xcite to train compact bert models in a task-specific and task-agnostic fashion respectively by mimicking the behavior of large teacher models. recently, @xcite proposed progressive module replacing, which trains a compact student model by progressively replacing the teacher layers with their more compact substitutes. ## combining the bag of tricks as we described in section 4.3, we discover that post-training quantization (ptq) can improve the robustness of a model while knowledge distillation (kd) loss benefits the loyalty of a compressed model. thus, by combining multiple compression techniques, we expect to achieve a higher speed-up ratio with improved accuracy, loyalty and robustness. to combine kd with other methods, we replace the original cross-entropy loss in quantizationaware training and module replacing with the knowledge distillation loss @xcite as in equation 2. for pruning, we perform knowledge distillation on the pruned model. we also apply the temperature re-scaling trick from (hinton @xcite with a fixed temperature of 10. as shown in ## conclusion in this paper, we propose label and probability loyalty to measure the correspondence of label and predicted probability distribution between compressed and original models. in addition to loyalty, we investigate the robustness of different compression techniques under adversarial attacks. these metrics reveal that post-training quantization and knowledge distillation can drastically improve robustness and loyalty, respectively. by combining multiple compression methods, we can further improve speed, accuracy, loyalty and robustness for various applications. our metrics help mitigate the gap between model training and deployment, shed light upon comprehensive evaluation for compression of pretrained language models, and call for the invention of new compression techniques.