On Attention Redundancy: A Comprehensive Study (2021)

Abstract: Multi-layer multi-head self-attention mechanism is widely applied in modern neural language models. Attention redundancy has been observed among attention heads but has not been deeply studied in the literature. Using BERT-base model as an example, this paper provides a comprehensive study on attention redundancy which is helpful for model interpretation and model compression. We analyze the attention redundancy with Five-Ws and How. (What) We define and focus the study on redundancy matrices generated from pre-trained and fine-tuned BERT-base model for GLUE datasets. (How) We use both token-based and sentence-based distance functions to measure the redundancy. (Where) Clear and similar redundancy patterns (cluster structure) are observed among attention heads. (When) Redundancy patterns are similar in both pre-training and fine-tuning phases. (Who) We discover that redundancy patterns are task-agnostic. Similar redundancy patterns even exist for randomly generated token sequences. (“Why”) We also evaluate influences of the pre-training dropout ratios on attention redundancy. Based on the phase-independent and task-agnostic attention redundancy patterns, we propose a simple zero-shot pruning method as a case study. Experiments on fine-tuning GLUE tasks verify its effectiveness. The comprehensive analyses on attention redundancy make model understanding and zero-shot model pruning promising.

## introduction multi-layer multi-head self-attention architectures (transformer @xcite ) are widely applied in modern language models, such as bert @xcite , roberta @xcite , openai gpt @xcite , gpt-2 @xcite and ernie2.0 @xcite , to name a few. redundancy phenomenon is discovered among attention heads. it demonstrates that many attention heads generate very similar attention matrices @xcite . we take the pre-trained bert-base model as an example. it learns 12-layer-12-head self-attention matrices describing dependencies between each pair of tokens in a sentence. then for each token, there are 144 attention vectors. we use jensen-shannon distance to measure the relationship between each pair of vectors. then for one sentence (consisting of a sequence of tokens), the token-averaged distance is utilized to imply the redundancy between each pair of attention matrices. smaller distance values reflect more redundancy. figure 1 shows the redundancy (distance) among 144 × 144 pairs of attention matrices averaged over 1000 randomly sampled sentences. we can see clear redundancy patterns (clusters with smaller distance areas) in consecutive attention layers. analyzing the attention redundancy helps to interpret the multi-layer multi-head self-attention architecture. various studies have attempted to re-veal the relationship among attention heads. examples are attention visualization @xcite , common attention patterns @xcite , attention head pruning @xcite , and probing test @xcite . existing works either focus on the 12 × 12 attention matrices and their effects on (pre-training or/and finetuning) performances or focus on linguistic features extracted by latent token vectors and attention matrices. though the redundancy phenomenon was discovered, no existing work studies the attention redundancy pattern itself (i.e., the 144 × 144 distance matrix in figure 1 ) deeply. this motivates us to conduct a comprehensive and complementary study on the attention redundancy phenomenon. in this paper, we take the bert-base model as a representative model to analyze the attention redundancy with five ws and how. as far as we know, many of the following discoveries are new to the research community. what is attention redundancy? given a distance function, we define the pairwise distance matrix (∈ r 144×144 ) of the 12 × 12 attention matrices of bert-base model as attention redundancy matrix. in this paper, we obtain redundancy matrices from both pre-trained and finetuned bert-base model for glue tasks as the research objects. how to measure attention redundancy? except for the two token-based measures, jensen-shannon distance @xcite and cosine similarity @xcite used in literature, we employ two more token-based distance function and three sentence-based ones to measure attention redundancy and analyze their similar redundancy patterns (please refer to section 4.1 for more details). the purpose is to alleviate the measuring bias of just using one distance function. sentence-based distances directly measure the relationship between two attention matrices without averaging over tokens. we visualize the redundancy patterns using various distance functions. where does attention redundancy exist? we find common hierarchical cluster structures in the set of token-based redundancy matrices and the set of sentence-based redundancy matrices, respectively. attention heads of earlier, middle, and deeper attention layers are clearly clustered in the redundancy matrices. we also demonstrate that highly correlated similar redundancy patterns exist in redundancy matrices generated based on different type of distances. when does attention redundancy occur? the redundancy is phase-independent. common redundancy patterns are discovered in both the pre-trained phase and fine-tuned phases. for any downstream task with any distance function, we notice highly correlated attention redundancy patterns between two phases. who (which task) has attention redundancy? we surprisingly realize that the redundancy is task-agnostic. the redundancy patterns are highly correlated across different tasks. we even randomly generate token sequences as input in the pre-trained bert-base model. very similar attention redundancy patterns occur as well. based on this astonishing discovery, as a case study application, we propose a simple zero-shot head-pruning strategy based on clustering results using redundancy matrices. compared to other complex pruning strategies, e.g., @xcite @xcite @xcite , the most important is that without knowing any data of fine-tuning tasks, this pruning can be effectively and efficiently conducted just based on some randomly generated token sequences with the pre-trained bert-base model. the only effort is to compute one or several redundancy matrices. results reflect that for most glue tasks, the proposed pruning strategy based on redundancy matrices can prune up to 75% to 85% of attention heads while keeping comparable fine-tuning performances. it's hard to tell the reason of the redundancy patterns (that's why we use the quoted "why"). however, we conduct experiments to evaluate the effects on attention redundancy of dropout ratios in the pre-training phase which are suspected as one of reasons @xcite . when we use sentence-based distance, a monotonic trend is found. attention heads tend to be more redundant when increasing dropout ratios. when we use token-based distances, a complex "n"-shape effect exists. we also notice that the redundancy is more sensitive to dropouts in hidden linear transformations than to dropouts in the self-attention mechanism. we believe that above-mentioned new findings in this paper make the redundancy analyses a promis-ing research direction in model interpretation and model compression, and so on. ## related work existing literature analyzes multi-layer multi-head self-attention architectures based language models (e.g., bert) from different aspects including word/token embedding latent space, linguistic knowledge interpretability, attention mechanism, and so on @xcite . the output of bert attention layers are token embedding vectors. one output vector for one token aggregates contextual information from the whole sentence. in the vector space, attention can produce strong representations for syntactic phenomena and phrasal information @xcite , but small improvements on semantic tasks @xcite . @xcite showed that contextualized representations of all words are not isotropic in any layer. @xcite revealed isotropy in the clustered contextual embedding space, and found low-dimensional manifolds. in the fine-tuning process, @xcite showed that the last two attention layers encode task-specific features while earlier layers capture more fundamental information. another set of works focus on the ability of extracting linguistic knowledge. bert can obtain syntactic dependencies, parts of speech tags, word disambiguation, and so forth @xcite @xcite . it showed that the same layer learns similar knowledge @xcite . positional information is encoded in bert lower layers @xcite . @xcite argued that middle and last attention layers can extract dependency relations and distant information, respectively. even with bert's success, it still struggles handling some linguistic information and tasks. bert does not excel at numbers, negation, inferences and role-based event prediction @xcite ). it's questionable to provide transparency or meaningful explanations for model predictions on downtream tasks @xcite . since self-attention is the fundamental mechanism in bert, existing works also investigate extracted attention vectors and/or matrices. these are most relevant to our study. @xcite and @xcite found some common attention patterns, such as patterns on delimiter tokens, block, and heterogeneous patterns. also, redundancy and overparameterization is discovered in attention heads. by using attention-based probing classifiers, @xcite showed that heads in the same layer often exhibits similar behaviors. attention heads can be pruned with different strategies but keep comparable performance in downstream tasks @xcite . some layers can even be reduced to a single head @xcite . understanding attention redundancy can help interpret pre-trained/fine-tuned language models and guide model compression. but no systematic study on attention redundancy exists. this paper provides a deep study (five-ws and how) on the attention redundancy. ## what: redundancy matrices in this paper, we use bert-base model as a representative to study the attention redundancy existing in multi-layer multi-head self-attention mechanisms. as a pre-trained language model, bert has been verified with outstanding fine-tuning performances on many downstream language understanding tasks. in bert-base model there are 12 self-attention layers each of which consists of 12 self-attention heads. for one input (sentence or sentence pair with special tokens [cls] and [sep]), we extract 12 × 12 attention matrices. the size of each matrix is n × n where n is the number of tokens after bert tokenization. given a metric (e.g., distance function) and extracted attention matrices, we can measure the relationship between each pair of the 144 attention matrices, for instance the 144 × 144 matrix in figure 1 . we define the pair-wise relationship among the 144 heads (i.e., attention matrices) as redundancy matrix. we consider that the smaller the distances among some attention heads are, the more attention redundancy exists in them. as for the studying objects, we use the wellknown natural language understanding benchmark glue task foot_0 @xcite as evaluation objects for the attention redundancy analysis. among glue, cola is for english sentence acceptability judgments. sst-2 and sts-b are sentiment analyses tasks. mrpc and qqp are for sentencepair similarity classification. mnli, qnli, and rte are natural language inference tasks. cola and sst-2 are single sentence tasks, while others are predicting relationship between a pair of sentences. except that sts-b is a regression task, in the bert framework, others are formed as classification tasks. in experiments, we randomly select 1000 data samples from each development set foot_2 as data instances and report the averaged results. we feed them to (pre-trained and fine-tuned) bert-base model to generate attention matrices. we use pytorch bert-base model as the pretrained model foot_3 @xcite . for finetuning, we train each task with suggested hyperparameters max-@xmath0, num-@xmath1, batchsize-per-@xmath2. all results in this paper are averaged over 5 trials. ## how: distance functions a fundamental question about attention redundancy is how to measure the similarity or distance between two attention matrices. the smaller the distance is, the more redundancy should exist. in this section, we introduce two levels of distance functions, token-based and sentence-based distances. the aim is to inspect their interpretabilities and consistency on quantifying the attention redundancy and to alleviate measuring bias of just using a certain distance function. let dist(a i , a j ) be a distance function to measure the relationship between the ith and jth attention matrices a i and a j . note that we reshape the 12 layers×12 heads into 144 attention matrices because the inputs of dist are two matrices. intuitively, there should be three properties in a defined distance function. so for the following distances, we modify them according to these requirements and normalize them into range [0,1] for easy comparisons. ## where: redundancy in layers in this section, we discuss the redundancy patterns in details and show that similar redundancy patterns exist in redundancy matrices when using the four token-based distance functions. so do for the three sentence-based distance functions. ## when: redundancy is phase-independent as a pre-trained language model, bert is finetuned for downstream tasks. in this section, we check the redundancy patterns in both pre-trained and fine-tuned bert-base model. figure 4 shows the attention redundancy matrices of fine-tuned bert-base for cola using the two levels of distances (results for other tasks are shown in appendix b). very similar redundancy patterns to those in figure 2 can be observed. to quantify the similarity, we simply compute the absolute values of differences between two corresponding redundancy matrices in figure 2 and figure 4 . the mean difference value is just 0.035. in addition, we compute correlations between pretrained and fine-tuned bert-base for each glue task under each distance function. as shown in figure 5 , high correlations are shown in pre-trained and fine-tuned phases for all glue tasks and all distance functions. to summarize, attention redundancy patterns in pre-trained and fine-tuned phases are very similar and highly correlated. ## who: redundancy is task-agnostic in this section, we analyze the attention redundancy from the task-oriented perspective. specifically, for each distance function and each phase, we compute the correlation between redundancy matrices of each pair of tasks. the correlation results for pre-trained bert-base model under different distances are shown in figure 6 (results for fine-tuned bert-base are presented in appendix d). we find that the redundancy patterns across tasks are very similar with each other. based on the task categories provided in @xcite , we can also notice that relatively higher correlations occur in the two single-sentence tasks (cola and sst-2), the three similarity and paraphrase tasks (mrpc, qqp, and sts-b), and the three inference tasks (mnli, qnli, and rte), respectively. to further check the task-agnostic observation, we even randomly generate 1000 token sequences with various lengths (uniformly distributed) as input for the pre-trained bert-base ("random" in figure 6 ). we can observe high correlations among redundancy patterns of glue datasets and random inputs. we conclude that attention redundancy patterns are task-agnostic. we think that it may be caused by the input formatting in bert. in fine-tuning, the special token [cls] and [sep] are inserted in the beginning and at the end of one input sentence, respectively. if there are two sentences as input, they are also delimited by the special token @xcite . it is found that [cls] and [sep] may dominate the attention distribution in some attention heads @xcite such that high similarities are observed through different tasks. based on this surprising discovery, we can apply this task-agnostic observation for pruning redundant attention heads. without knowing any data of downstream task, we conduct a simple but effective clustering-based zero-shot pruning strategy based on attention redundancy in the following. ## "why": effects of pre-training dropouts it is hard to answer the fundamental question why attention redundancy exists. however, in this section, we examine one factor that may be the cause. dropout in the pre-training phase is suspected to be one reason resulting in redundant attentions @xcite . @xcite didn't evaluate that. in this section, we investigate effects of various dropout ratios in the pre-training process on the attention redundancy. in bert pretraining, there are two dropout ratios, attentionhead-dropout-ratio and hidden-dropout-ratio. the former randomly deactivates a ratio of trainable weights in the attention head (key, query, and value transformation matrices). the latter deactivates some weights in the linear transformation matrices which integrate all attention heads after each attention layer. their default values are 0.1. we manually set various values (from 0.0 to 0.9) for those two dropout ratios and train a bert masked language model on wiki103 training dataset foot_5 from scratch. we train 100 epochs or until training loss converges. then we randomly select 1000 sentences from the test dataset as objects. for each sentence, we calculate the mean value of attention redundancy matrix using js and dcor distance. the final reported values are averaged over the 1000 sentences. results of three settings are shown in figure 9 . "only attn" means that we only change the attention-head-dropout-ratio and keep the hiddendropout-ratio as the default value 0.1. "only hidden" is changing hidden-dropout-ratio and keeping attention-head-dropout-ratio as default. "attn & hidden" modifies both dropout ratios. in both figures, smaller distance values reflect heavier attention redundancy. we can see that the token-based measure js and sentence-based measure dcor show reversed trends in the middle range ([0,2 0.7]). for js on all three settings, when dropout ratio increases (i.e., more inactive weights updating in pre-training) the distances first drop down to the lowest values (when dropout ratio is 0.2) and increase and then drop again heavily. on the other hand, for dcor, the distance decreases (heavier attention redundancy) along the increased dropout ratio. we conclude that dropout ratio does not play a simple effect on the attention redundancy. the "n"-shape effects shown in the left figure (tokenbased distance js) is demonstrated from the view of attention vectors' similarity. on the other hand, the sentence-based distance (dcor) figure shows that higher dropout ratios result in more redundancy 10 . in both figures, we observe that slopes of "attn & hidden" and "only hidden" are steeper than that of "only attn". this means that dropout in the hidden transformations affects the attention redundancy more than the attention dropout. there is no doubt that more factors must affect the attention redundancy. we leave the "why" in future study. ## conclusion using bert-base model as an example, we comprehensively investigated the attention redundancy in multi-layer multi-head self-attention based language models. the redundancy was measured by distance functions at token level and sentence level. at both levels, we found that many heads are not distinct from each other, and clear clustering effects were observed. we discovered that the attention redundancy is phase-independent and taskagnostic. specifically, compared to a pre-trained model, the redundancy patterns do not change much after fine-tuning on multiple downstream tasks. we also shown complex influences on redundancy of dropout ratios in hidden transformations and self-attention. based on these discoveries, we design a zero-shot strategy to prune attention heads. compared to existing methods, the zero-shot pruning is simple and robust (task-agnostic). in the near future, we are interested in experimenting this method over more self-attention based pre-trained language models and more downstream tasks. this is the appendix for naacl-hlt 2021 paper: yuchen bian, jiaji huang, xingyu cai, jiahong yuan, and kenneth church. on attention redundancy: a comprehensive study.