  Transformer-based pre-trained language models have significantly improved the
performance of various natural language processing (NLP) tasks in the recent
years. While effective and prevalent, these models are usually prohibitively
large for resource-limited deployment scenarios. A thread of research has thus
been working on applying network pruning techniques under the
pretrain-then-finetune paradigm widely adopted in NLP. However, the existing
pruning results on benchmark transformers, such as BERT, are not as remarkable
as the pruning results in the literature of convolutional neural networks
(CNNs). In particular, common wisdom in pruning CNN states that sparse pruning
technique compresses a model more than that obtained by reducing number of
channels and layers (Elsen et al., 2020; Zhu and Gupta, 2017), while existing
works on sparse pruning of BERT yields inferior results than its small-dense
counterparts such as TinyBERT (Jiao et al., 2020). In this work, we aim to fill
this gap by studying how knowledge are transferred and lost during the
pre-train, fine-tune, and pruning process, and proposing a knowledge-aware
sparse pruning process that achieves significantly superior results than
existing literature. We show for the first time that sparse pruning compresses
a BERT model significantly more than reducing its number of channels and
layers. Experiments on multiple data sets of GLUE benchmark show that our
method outperforms the leading competitors with a 20-times weight/FLOPs
compression and neglectable loss in prediction accuracy.
