Self-attention based models have achieved remarkable success in natural language processing . However, the self-attention network design is questioned as suboptimal in recent studies, due to its veiled validity and high redundancy . In this paper, we focus on pre-trained language models with self-pruning training design on task-specific tuning. We demonstrate that the lighter state-of-the-art models with nearly 80\% of self-attention layers pruned, may achieve even better resultson multiple tasks, including natural language understanding, document classification, named entity recognition and POS tagging, with nearly twice faster inference .
Beyond the success story of pre-trained language models (PrLMs) in recent natural language processing , they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves as a therapy. However, existing methods like random-based, knowledge-based and search-based dropout are more general but less effective onto self-attention based models, which are broadly chosen as the fundamental architecture of PrLMs . In this paper, we propose a novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust task-specific tuning. We demonstrate that state-of-the-art models with elaborate training design may achieve much stronger results. We verify the universality of our approach on extensive natural language processing tasks .
R
Self-attention based models have achieved remarkable success in
Beyond the success story of pre-trained language models (PrLMs) in recent
R
. However, the
, they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves as a therapy. However, existing methods like random-based, knowledge-based and search-based dropout are more general but less effective onto
R
network design is questioned as suboptimal in recent studies, due to its veiled validity and high redundancy
based models, which are broadly chosen as the fundamental architecture of PrLMs
R
focus on pre-trained language models with self-pruning training design on
propose a novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust
D
the lighter
R
nearly 80\% of self-attention layers pruned, may achieve even better resultson multiple tasks, including natural language understanding, document classification, named entity recognition and POS tagging, with nearly twice faster inference
elaborate training design may achieve much stronger results. We verify the universality of our approach on extensive natural language processing tasks