Weight-Inherited Distillation for Task-Agnostic BERT Compression (2024)

Abstract: Knowledge Distillation (KD) is a predominant approach for BERT compression.Previous KD-based methods focus on designing extra alignment losses for the student model to mimic the behavior of the teacher model.These methods transfer the knowledge in an indirect way.In this paper, we propose a novel Weight-Inherited Distillation (WID), which directly transfers knowledge from the teacher.WID does not require any additional alignment loss and trains a compact student by inheriting the weights, showing a new perspective of knowledge distillation.Specifically, we design the row compactors and column compactors as mappings and then compress the weights via structural re-parameterization.Experimental results on the GLUE and SQuAD benchmarks show that WID outperforms previous state-of-the-art KD-based baselines.Further analysis indicates that WID can also learn the attention patterns from the teacher model without any alignment loss on attention distributions.The code is available at https://github.com/wutaiqiang/WID-NAACL2024.

## introduction transformer-based pre-trained language models (plms), such as bert @xcite , roberta @xcite , xlnet @xcite , have achieved great success in many natural language process (nlp) tasks. these models are pre-trained on massive corpus via self-supervised tasks to learn contextualized text representations. however, plms have high costs in terms of storage, memory, and computation time, which brings challenges to online services in real-life applications. therefore, it is crucial and feasible to compress plms while maintaining their performance. knowledge distillation (kd), which trains a compact student model by mimicking the behavior of a teacher model, is a predominant method ## conclusion this work proposes a novel weight-inherited distillation (wid) method for task-agnostic bert compression. in wid, we factorize the compression process as weight mappings, and then design the row compactors and column compactors for row mappings and column mappings, respectively. empirical results on various student model sizes 20 demonstrate the effectiveness of wid. further analysis indicates that inheriting the weights can also inherit high-level semantic knowledge such as attention patterns. in future work, we would consider reducing the extra memory cost by compactor layers, such as compactor sharing. moreover, employing wid on the large language model (llm) would be another interesting topic.