I ncorporating R esidual and N ormalization L ayers into A nalysis of M asked L anguage M odels (2021)

Abstract: Transformer architecture has become ubiquitous in the natural language processing field. To interpret the Transformer-based models, their attention patterns have been extensively analyzed. However, the Transformer architecture is not only composed of the multi-head attention; other components can also contribute to Transformers’ progressive performance. In this study, we extended the scope of the analysis of Transformers from solely the attention patterns to the whole attention block, i.e., multi-head attention, residual connection, and layer normalization. Our analysis of Transformer-based masked language models shows that the token-to-token interaction performed via attention has less impact on the intermediate representations than previously assumed. These results provide new intuitive explanations of existing reports; for example, discarding the learned attention patterns tends not to adversely affect the performance. The codes of our experiments are publicly available.

## introduction transformer architecture @xcite has advanced the state of the art in a wide range of natural language processing (nlp) tasks @xcite @xcite . along with this, transformers have become a major subject of research from the viewpoints of engineering @xcite and scientific studies @xcite . in particular, the multi-head attention, a core component of transformers, has been extensively analyzed @xcite @xcite @xcite @xcite . while these analyses suggest that the multi-head attention contributes (a) existing analysis focusing only on the multi-head attention @xcite . figure 1 : visualizations of the token-by-token interactions in each layer when a sentence pair is fed into the pre-trained bert-base. the diagonal elements correspond to the effect of preserving the original input information. the contrast between figures 1a and 1b demonstrates that the contextual information contributed less to the computation of the output representations than previously expected. to capturing linguistic information such as semantic and syntactic relations, some reports question the importance of attention. for example, several studies in fields ranging from nlp @xcite to neuroscience @xcite empirically found that discarding learned attention patterns from transformers retains or even improves their performance in downstream tasks and the ability to simulate human brain activity. these observations imply that transformers do not heavily rely on the multi-head attention alone, and the other components contribute to their progressive performance. in this study, we broaden the scope of the analysis from the multi-head attention to the whole attention block, i.e., the multi-head attention, residual connection, and layer normalization. our analysis of the transformer-based masked language models @xcite revealed that the newly incorporated components have a larger impact than expected in previous studies @xcite (figure 1 ). more concretely, we introduce an exact decomposition of the operations in the whole attention block exploiting the norm-based analysis @xcite . our analysis quantifies the impact of the two contrasting effects of the attention block : (i) "mixing" the input representations via attention and (ii) "preserving" the original input mainly via residual connection (section 3). our analysis reveals that the preserving effect is more dominant in each attention block than previously estimated @xcite . the results also reveal the detailed mechanism of each component in the attention block. the residual connections pass through much larger vectors than the vectors produced by the multi-head attention. the layer normalization also reduces the effect of the operation via attention. our finding of the relatively small impact of the multi-head attention provides new intuitive interpretations for some existing reports, for example, discarding the learned attention patterns did not adversely affect their performance. our analysis also provides a new intuitive perspective on the behaviors of transformer-based masked language models. for example, bert @xcite highlights low-frequency (informative) words in encoding texts, which is consistent with the existing methods for effectively computing text representations @xcite . the contributions of this study are as follows: ## background: transformer architecture the transformer architecture consists of a stack of layers. each layer has an attention block, which is responsible for capturing the interactions between input tokens. the attention block can be further divided into the three components: multi-head attention (attn), residual connection (res), and layer normalization (ln) (figure 2 ). this block can be written as the following composite function: ## proposal: analyzing attention blocks for analyzing transformers, solely observing the attention weights has been a common method @xcite . we extend the scope of analysis to the whole attention block (attn, res, and ln). ## experiments: analysis of mixing ratio the context-mixing ratio of the attention blocks in pre-trained masked language models was analyzed using the proposed norm-based analysis. the obtained results were different from those of the existing methods that analyze only some of the components in the attention block. ## detailed analysis we further analyzed the mixing ratio of the masked language models in detail from the perspectives of both the layer and word attributes. in this section, we inherit the experimental setup (section 4.1) from the previous section and demonstrate results for bert-base with the wikipedia dataset. the results for the other experimental settings are shown in appendix b and d. note that only the finding reported in section 5.2 did not generalize across model variants, and we exceptionally discuss this point in the body. ## conclusions in this paper, we have extended a norm-based analysis to broaden the scope of analyzing transformers from the multi-head attention alone to the whole attention block, i.e., multi-head attention, residual connection, and layer normalization. our analysis of the masked language models revealed that the context-mixing ratio in each block is much lower than expected in previous studies, demonstrating that res and ln largely cancel the mixing by attn. this observation can provide new explanations for some unexpected results were reported on transformers in fields ranging from nlp to neuroscience (e.g., discarding the learned attention patterns did not adversely affect the performance). our detailed analysis further suggested that bert discounts highly frequent, low-informative tokens. although our method is applicable to analyzing other variants of transformers, our experiments were limited to the transformer-based masked language models. in addition, the transformer is not composed of only the attention block; feed-forward and embedding layers also exist. we plan to extend this work in both directions.