Rethinking the Value of Transformer Components (2020)

Abstract: Transformer becomes the state-of-the-art translation model, while it is not well studied how each intermediate component contributes to the model performance, which poses significant challenges for designing optimal architectures. In this work, we bridge this gap by evaluating the impact of individual component (sub-layer) in trained Transformer models from different perspectives. Experimental results across language pairs, training strategies, and model capacities show that certain components are consistently more important than the others. We also report a number of interesting findings that might help humans better analyze, understand and improve Transformer models. Based on these observations, we further propose a new training strategy that can improves translation performance by distinguishing the unimportant components in training.

## introduction transformer @xcite has achieved the state-of-the-art performance on a variety of translation tasks. it consists of different stacked components, including self-attention, encoder-attention, and feed-forward layers. however, so far not much is known about the internal properties and functionalities it learns to achieve the performance, which poses significant challenges for designing optimal architectures. in this work, we bridge the gap by conducting a granular analysis of components on trained transformer models. we attempt to understand how does each component contribute to the model outputs. specifically, we explore two metrics to evaluate the impact of a particular component on the model performance: 1) contribution in information flow that manually masks individual component each time and evaluate the performance without that component; and 2) criticality in representation generalization that depends on how much closer the weights can get for each component to the initial weights while still maintaining performance. those two metrics evaluate the component importance of a trained transformer model from different perspectives. empirical results on two benchmarking datasets reveal the following observations ( §3.1): the findings are consistent across different evaluation metrics, translation datasets, initialization seeds and model capacities, demonstrating their robustness. we further analyze the underlying reason ( §3.2), and find that lower dropout ratio and more training data lead to less unimportant components. unimportant components can be identified at early stage of training, which are not due to deficient training. finally, we show that unimportant components can be rewound @xcite to further improve the translation performance of transformer models ( §3.3). ## methodology in this section, we evaluate the importance of individual transformer components via two different perspective: contribution in information flow and criticality in representation generalization. the ultimate goal of machine translation is to fully transform the information from the source side to the target side. it is essential to understand how information flows from the input, across the encoder and the decoder, to the output. figure 1 shows an example to illustrate how information flows across a basic transformer component (i.e., residual sub-layer). ## experiments data and setup we conducted experiments on the benchmarking wmt2014 english-german (en-de) and english-french (en-fr) translation datasets, which consist of 4.6m and 35.5m sentence pairs respectively. we employed bpe @xcite with 32k merge operations for both language pairs, and used case-sensitive 4-gram nist bleu score @xcite as our evaluation metric. unless otherwise stated, the transformer model consists of 6-layer encoder and decoder. the layer size is 512, the size of feed-forward sub-layer is 2048, and the number of attention heads is 8. we followed the settings in @xcite to train the transformer models on the en-de and en-fr datasets. we set the dropout as 0.1 and the initialization seed as 1 for all transformer models . ## conclusion in this work we investigate the impact of individual components in transformer on model performance. experimental results in a couple of settings show that different components are not equally important. the decoder self-attention layers are least important, and the decoder feed-forward layers are most important. the components that are closer to the model input and output are consistently more important than the others. upper encoder-attention layers in decoder are more important than lower encoder-attention layers. further in-depth analyses reveal that the dropout and training data size can affect the number of unimportant components. we also find that unimportant components can be identified at early stage of training and their existence is not because of deficient training. finally, we show that rewinding the unimportant components and then fine-tuning the transformer model for a few more steps can further improve translation performance. future directions include designing better approaches to evaluate the impact of components (e.g., from the perspective of information flow), and validating our findings on other nmt architectures such as rnmt @xcite and convs2s @xcite .