The attention mechanism is a key component of the neural revolution in Natural Language Processing (NLP). As the size of attention-based models has been scaling with the available computational resources, a number of pruning techniques have been developed to detect and to exploit sparseness in such models in order to make them more efficient. The majority of such efforts have focused on looking for attention patterns and then hard-coding them to achieve sparseness, or pruning the weights of the attention mechanisms based on statistical information from the training data. In this paper , we marry these two lines of research by proposing Attention Pruning (AP): a novel pruning framework that collects observations about the attention patterns in a fixed dataset and then induces a global sparseness mask for the model. Through attention pruning, we find that about 90\% of the attention computation can be reduced for language modelling and about 50\% for machine translation and \%natural language inference prediction with BERT on GLUE tasks, while maintaining the quality of the results. Additionally , using our method, we discovered important distinctions between self- and cross-attention patterns, which could guide future NLP research in attention-based modelling. Our approach could help develop better models for existing or for new NLP applications , and generally for any model that relies on attention mechanisms . Our implementation and instructions to reproduce the experiments are available at URL
The attention mechanism is a key component of the neural revolution in Natural Language Processing (NLP). As the size of attention-based models has been scaling with the available computational resources, a number of pruning techniques have been developed to detect and to exploit sparseness in such models in order to make them more efficient. The majority of such efforts have focused on looking for attention patterns and then hard-coding them to achieve sparseness, or pruning the weights of the attention mechanisms based on statistical information from the training data. Here , we marry these two lines of research by proposing Attention Pruning (AP): a novel pruning framework that collects observations about the attention patterns in a fixed dataset and then induces a global sparseness mask for the model. This can save 90\% of the attention computation for language modelling and about 50\% for machine translation and for solving GLUE tasks, while maintaining the quality of the results. Moreover , using our method, we discovered important distinctions between self- and cross-attention patterns, which could guide future NLP research in attention-based modelling. Our framework can in principle speed up any model that uses attention mechanism, thus helping develop better models for existing or for new NLP applications . Our implementation is available at URL
R
In this paper
Here
R
Through attention pruning, we find that about
This can save
D
can be reduced
R
\%natural language inference prediction with BERT on
for solving
R
Additionally
Moreover
R
approach could help
framework can in principle speed up any model that uses attention mechanism, thus helping
D
, and generally for any model that relies on attention mechanisms
R
and instructions to reproduce the experiments are
is