Attention is Not Only a Weight: Analyzing Transformers with Vector Norms (2020)

Abstract: Attention is a key component of Transformers, which have recently achieved considerable success in natural language processing. Hence, attention is being extensively studied to investigate various linguistic capabilities of Transformers, focusing on analyzing the parallels between attention weights and specific linguistic phenomena. This paper shows that attention weights alone are only one of the two factors that determine the output of attention and proposes a norm-based analysis that incorporates the second factor, the norm of the transformed input vectors. The findings of our norm-based analyses of BERT and a Transformer-based neural machine translation system include the following: (i) contrary to previous studies, BERT pays poor attention to special tokens, and (ii) reasonable word alignment can be extracted from attention mechanisms of Transformer. These findings provide insights into the inner workings of Transformers.

## introduction transformers @xcite @xcite @xcite have improved the state-of-the-art in a wide range of natural language processing tasks. the success of the models has not yet been sufficiently explained; hence, substantial research has focused on assessing the linguistic capabilities of these models @xcite . one of the main features of transformers is that they utilize an attention mechanism without the use of recurrent or convolutional layers. the attention mechanism computes an output vector by accumulating relevant information from a sequence of input vectors. specifically, it assigns attention weights (i.e., relevance) to each input, and sums up input vectors based on their weights. the analysis of correlations between attention weights and various linguistic phenomena (i.e., weight-based analysis) is a prominent research area @xcite @xcite @xcite @xcite . this paper first shows that weight-based analysis is insufficient to analyze the attention mechanism. weight-based analysis is a common approach to analyze the attention mechanism by simply tracking attention weights. the attention mechanism can be expressed as a weighted sum of linearly transformed vectors (section 2.2); however, the effect of transformed vectors in weightbased analysis is ignored. we propose a normbased analysis that considers the previously ignored factors (section 3). in this analysis, we measure the norms (lengths) of the vectors that were summed to compute the output vector of the attention mechanism. using the norm-based analysis of bert (section 4), we interpreted the internal workings of the model in more detail than when weight-based analysis was used. for example, the weight-based analysis @xcite reports that specific tokens, such as periods, commas, and special tokens (e.g., separator token; @xcite ), tend to have high attention weights. however, our norm-based analysis found that the information collected from vectors corresponding to special tokens was considerably lesser than that reported in the weight-based analysis, and the large attention weights of these vectors were canceled by other factors. additionally, we found that bert controlled the levels of contribution from frequent, less informative words by controlling the norms of their vectors. in the analysis of a transformer-based nmt system (section 5), we reinvestigated how accurate word alignment can be extracted from the source-target attention. the weight-based results of @xcite , @xcite , @xcite have empirically shown that word alignments induced by the source-target attention of the transformer-based nmt systems are noisy. our experiments show that more accurate alignments can be extracted by focusing on the vector norms. the contributions of this study are as follows: • we propose a novel method of analyzing an attention mechanism based on vector norms (norm-based analysis). the method considers attention weights and previously ignored factors, i.e., the norm of the transformed vector. ## proposal: norm as a degree of attention as described in section 2.3, analyzing the attention mechanism with only attention weights neglects the effect of the transformed vector f (x j ), which has a significant impact as we discussed later. herein, we propose the measurement of the norm of the weighted transformed vector αf (x) , given by equation 3 , to analyze the attention mechanism behavior. foot_0 unlike in previous studies, we analyzed the behaviors of the norms, αf (x) and f (x) , and α to gain more in-depth insights into the functioning of attention. the proposed method of analyzing the attention mechanism is called norm-based analysis and the method that solely analyzes the attention weights is called weight-based analysis. in sections 4 and 5, we provide insights into the working of transformers using norm-based analysis. appendix a explains that our norm-based analysis can also be effectively applied to an entire multi-head attention mechanism. general settings: following the previous studies @xcite @xcite @xcite , we used the pre-trained bert-base foot_1 , with 12 layers, each containing 12 attention heads. we used the data provided by @xcite for the analysis. 5 the data contains 992 sequences extracted from wikipedia, where each sequence consists of two consecutive paragraphs, in the form of: [cls] paragraph1 [sep] paragraph2 @xcite . each sequence consists of up to 128 tokens, with an average of 122 tokens. ## conclusions and future work this paper showed that attention weights alone are only one of two factors that determine the output of attention. we proposed the incorporation of another factor, the transformed input vectors. using our norm-based method, we provided a more detailed interpretation of the inner workings of transformers, compared to the studies using the weight-based analysis. we hope that this paper will inspire researchers to have a broader view of the possible methodological choices for analyzing the behavior of transformer-based models. we believe that these findings can provide insights not only into the interpretation of the behaviors of blackbox nlp systems but also into developing a more sophisticated transformer-based system. one possible direction is to design an attention mechanism that can collect almost no information from an input sequence as the current systems achieve it by exploiting the [sep] token. in future work, we plan to apply our norm-based analysis to attention in other models, such as finetuned bert, roberta @xcite , and al-bert @xcite . furthermore, we expect to extend the scope of analysis from the attention to an entire transformer architecture to better understand the inner workings and linguistic capabilities of the current powerful systems in nlp.