Multi-layer multi-head self-attention mechanism is widely applied in modern neural language models . Attention redundancy has been observed among attention heads but has not been deeply studied in the literature . Using BERT-base model as an example , this paper provides a comprehensive study on attention redundancy which is helpful for model interpretation and model compression . We analyze the attention redundancy with Five-Ws and How . ( What ) We define and focus the study on redundancy matrices generated from pre-trained and fine-tuned BERT-base model for GLUE datasets . ( How ) We use both token-based and sentence-based distance functions to measure the redundancy . ( Where ) Clear and similar redundancy patterns ( cluster structure ) are observed among attention heads . ( When ) Redundancy patterns are similar in both pre-training and fine-tuning phases . ( Who ) We discover that redundancy patterns are task-agnostic . Similar redundancy patterns even exist for randomly generated token sequences . ( " Why " ) We also evaluate influences of the pre-training dropout ratios on attention redundancy . Based on the phaseindependent and task-agnostic attention redundancy patterns , we propose a simple zero-shot pruning method as a case study . Experiments on fine-tuning GLUE tasks verify its effectiveness . The comprehensive analyses on attention redundancy make model understanding and zero-shot model pruning promising . Multi-layer multi-head self-attention architectures ( Transformer ( Vaswani et al . , 2017 ) ) are widely applied in modern language models , such as BERT ( Devlin et al . , 2019 ) , RoBERTa ( Liu et al . , 2019 ) , OpenAI GPT ( Radford et al . , 2018 ) , GPT-2 ( Radford et al . , 2019 ) and ERNIE2.0 ( Sun et al . , 2019 ) , to name a few . Redundancy phenomenon is discovered among attention heads . It demonstrates that many attention heads generate very similar attention matrices ( Clark et al . , 2019 ; Kovaleva et al . , 2019 ) . We take the pre-trained BERT-base model as an example . It learns 12-layer-12-head self-attention matrices describing dependencies between each pair of tokens in a sentence . Then for each token , there are 144 attention vectors . We use Jensen-Shannon distance to measure the relationship between each pair of vectors . Then for one sentence ( consisting of a sequence of tokens ) , the token-averaged distance is utilized to imply the redundancy between each pair of attention matrices . Smaller distance values reflect more redundancy . Figure 1 shows the redundancy ( distance ) among 144 × 144 pairs of attention matrices averaged over 1000 randomly sampled sentences . We can see clear redundancy patterns ( clusters with smaller distance areas ) in consecutive attention layers . Analyzing the attention redundancy helps to interpret the multi-layer multi-head self-attention architecture . Various studies have attempted to re-veal the relationship among attention heads . Examples are attention visualization ( Vig and Belinkov , 2019 ) , common attention patterns ( Kovaleva et al . , 2019 ) , attention head pruning ( Voita et al . , 2019 ) , and probing test ( Clark et al . , 2019 ) . Existing works either focus on the 12 × 12 attention matrices and their effects on ( pre-training or / and finetuning ) performances or focus on linguistic features extracted by latent token vectors and attention matrices . Though the redundancy phenomenon was discovered , no existing work studies the attention redundancy pattern itself ( i.e. , the 144 × 144 distance matrix in Figure 1 ) deeply . This motivates us to conduct a comprehensive and complementary study on the attention redundancy phenomenon . In this paper , we take the BERT-base model as a representative model to analyze the attention redundancy with Five Ws and How . As far as we know , many of the following discoveries are new to the research community . What is attention redundancy ? Given a distance function , we define the pairwise distance matrix ( ∈ R 144×144 ) of the 12 × 12 attention matrices of BERT-base model as attention redundancy matrix . In this paper , we obtain redundancy matrices from both pre-trained and finetuned BERT-base model for GLUE tasks as the research objects . How to measure attention redundancy ? Except for the two token-based measures , Jensen-Shannon distance ( Clark et al . , 2019 ) and cosine similarity ( Kovaleva et al . , 2019 ) used in literature , we employ two more token-based distance function and three sentence-based ones to measure attention redundancy and analyze their similar redundancy patterns ( please refer to Section 4.1 for more details ) . The purpose is to alleviate the measuring bias of just using one distance function . Sentence-based distances directly measure the relationship between two attention matrices without averaging over tokens . We visualize the redundancy patterns using various distance functions . Where does attention redundancy exist ? We find common hierarchical cluster structures in the set of token-based redundancy matrices and the set of sentence-based redundancy matrices , respectively . Attention heads of earlier , middle , and deeper attention layers are clearly clustered in the redundancy matrices . We also demonstrate that highly correlated similar redundancy patterns exist in redundancy matrices generated based on different type of distances . When does attention redundancy occur ? The redundancy is phase-independent . Common redundancy patterns are discovered in both the pre-trained phase and fine-tuned phases . For any downstream task with any distance function , we notice highly correlated attention redundancy patterns between two phases . Who ( which task ) has attention redundancy ? We surprisingly realize that the redundancy is task-agnostic . The redundancy patterns are highly correlated across different tasks . We even randomly generate token sequences as input in the pre-trained BERT-base model . Very similar attention redundancy patterns occur as well . Based on this astonishing discovery , as a case study application , we propose a simple zero-shot head-pruning strategy based on clustering results using redundancy matrices . Compared to other complex pruning strategies , e.g. , ( Tang et al . , 2019 ; Jiao et al . , 2019 ; Fan et al . , 2019 ; Wang et al . , 2019 ; McCarley , 2019 ) , the most important is that without knowing any data of fine-tuning tasks , this pruning can be effectively and efficiently conducted just based on some randomly generated token sequences with the pre-trained BERT-base model . The only effort is to compute one or several redundancy matrices . Results reflect that for most GLUE tasks , the proposed pruning strategy based on redundancy matrices can prune up to 75 % to 85 % of attention heads while keeping comparable fine-tuning performances . " Why " does the phase-independent and taskagnostic attention redundancy happen ? It 's hard to tell the reason of the redundancy patterns ( that 's why we use the quoted " Why " ) . However , we conduct experiments to evaluate the effects on attention redundancy of dropout ratios in the pre-training phase which are suspected as one of reasons ( Clark et al . , 2019 ) . When we use sentence-based distance , a monotonic trend is found . Attention heads tend to be more redundant when increasing dropout ratios . When we use token-based distances , a complex " N"-shape effect exists . We also notice that the redundancy is more sensitive to dropouts in hidden linear transformations than to dropouts in the self-attention mechanism . We believe that above-mentioned new findings in this paper make the redundancy analyses a promis-ing research direction in model interpretation and model compression , and so on . Using BERT-base model as an example , we comprehensively investigated the attention redundancy in multi-layer multi-head self-attention based language models . The redundancy was measured by distance functions at token level and sentence level . At both levels , we found that many heads are not distinct from each other , and clear clustering effects were observed . We discovered that the attention redundancy is phase-independent and taskagnostic . Specifically , compared to a pre-trained model , the redundancy patterns do not change much after fine-tuning on multiple downstream tasks . We also shown complex influences on redundancy of dropout ratios in hidden transformations and self-attention . Based on these discoveries , we design a zero-shot strategy to prune attention heads . Compared to existing methods , the zero-shot pruning is simple and robust ( task-agnostic ) . In the near future , we are interested in experimenting this method over more self-attention based pre-trained language models and more downstream tasks . This is the appendix for NAACL-HLT 2021 paper : Yuchen Bian , Jiaji Huang , Xingyu Cai , Jiahong Yuan , and Kenneth Church . On Attention Redundancy : A Comprehensive Study .

Challenge: While there are works that report a redundancy among attention heads in modern language models, no works investigate its pattern deeply.
Approach: They perform token and sentence level analysis on redundancy matrices from pre-trained and fine-tuned BERT-base models and further propose a pruning method based on findings.
Outcome: They find that many heads are redundant regardless of phase and task, and show the proposed pruning method can perform robustly.