L ada BERT : Lightweight Adaptation of BERT through Hybrid Model Compression (2020)

Abstract: BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks. However, a major blocking issue of applying BERT to online services is that it is memory-intensive and leads to unsatisfactory latency of user requests, raising the necessity of model compression. Existing solutions leverage the knowledge distillation framework to learn a smaller model that imitates the behaviors of BERT. However, the training procedure of knowledge distillation is expensive itself as it requires sufficient training data to imitate the teacher model. In this paper, we address this issue by proposing a tailored solution named LadaBERT (Lightweight adaptation of BERT through hybrid model compression), which combines the advantages of different model compression methods, including weight pruning, matrix factorization and knowledge distillation. LadaBERT achieves state-of-the-art accuracy on various public datasets while the training overheads can be reduced by an order of magnitude.

## introduction the pre-trained language model, bert @xcite has led to a big breakthrough in various kinds of natural language understanding tasks. ideally, people can start from a pre-trained bert checkpoint and fine-tune it on a specific downstream task. however, the original bert models are memoryexhaustive and latency-prohibitive to be served in embedded devices or cpu-based online environments. as the memory and latency constraints vary in different scenarios, the pre-trained bert model should be adaptive to different requirements with accuracy retained to the largest extent. existing bert-oriented model compression solutions largely depend on knowledge distillation @xcite , which is inefficient and resource-consuming because a large training corpus is required to learn the behaviors of a teacher. for example, distilbert @xcite is re-trained on the same corpus as pre-training a vanilla bert from scratch; and tinybert @xcite utilizes expensive data augmentation to fit the distillation target. the costs of these model compression methods are as large as pre-training, which are unaffordable for low-resource settings. therefore, it is straight-forward to ask, can we design a lightweight method to generate adaptive models with comparable accuracy using significantly less time and resource consumption? in this paper, we propose ladabert (lightweight adaptation of bert through hybrid model compression) to tackle this problem. specifically, ladabert is based on an iterative hybrid model compression framework consisting of weighting pruning, matrix factorization and knowledge distillation. initially, the architecture and weights of student model are inherited from the bert teacher. in each iteration, the student model is first compressed by a small ratio based on weight pruning and matrix factorization, and is then fine-tuned under the guidance of teacher model through knowledge distillation. because weight pruning and matrix factorization help to generate better initial and intermediate status for knowledge distillation, both accuracy and efficiency can be greatly improved. we conduct extensive experiments on five public datasets of natural language understanding. as an example, the performance comparison of ladabert and state-of-the-art models on mnli-m dataset is illustrated in figure 1 . we can see that ladabert outperforms other bert-oriented model compression baselines at various model compression ratios. especially, ladabert outperforms bert-pkd significantly under 2.5× compression ratio and outperforms tinybert under 7.5× compression ratio while the training speed is accelerated by an order of magnitude. the rest of this paper is organized as follows. first, we summarize the related works of model compression and their applications to bert in section 2. then, the methodology of ladabert is introduced in section 3, and experimental results are presented in section 4. at last, we conclude this work and discuss future works in section 5. ## related work deep neural networks (dnns) have achieved great success in many areas in recent years, but the memory consumption and computational cost expand greatly with the growing complexity of models. thus, model compression has become an indispensable technique in practice, especially for low-resource scenarios. here we review the current progresses of model compression techniques briefly, and present their application to pre-trained bert models. ## conclusion model compression is a common way to deal with latency-critical or memory-intensive scenarios. existing model compression methods for bert are expansive as they require re-training on a large corpus to reserve the original performance. in this paper, we propose ladabert, a lightweight model compression pipeline that generates an adaptive bert model efficiently based on a given task and specific constraint. it is based on a hybrid solution, which conducts matrix factorization, weight pruning and knowledge distillation in a collaborative fashion. the experimental results demonstrate that ladabert is able to achieve comparable performance with other state-of-the-art solutions using much less training data and computation budget. therefore, ladabert can be easily plugged into various applications to achieve competitive performances with little training overheads. in the future, we would like to apply ladabert to large-scale industrial applications, such as search relevance and query recommendation.