  Self-attention based models have achieved remarkable success in natural
language processing. However, the self-attention network design is questioned
as suboptimal in recent studies, due to its veiled validity and high
redundancy. In this paper, we focus on pre-trained language models with
self-pruning training design on task-specific tuning. We demonstrate that the
lighter state-of-the-art models with nearly 80% of self-attention layers
pruned, may achieve even better results on multiple tasks, including natural
language understanding, document classification, named entity recognition and
POS tagging, with nearly twice faster inference.
