On the Sub-layer Functionalities of Transformer Decoder (2020)

Abstract: There have been significant efforts to interpret the encoder of Transformer-based encoder-decoder architectures for neural machine translation (NMT); meanwhile, the decoder remains largely unexamined despite its critical role. During translation, the decoder must predict output tokens by considering both the source-language text from the encoder and the target-language prefix produced in previous steps. In this work, we study how Transformer-based decoders leverage information from the source and target languages – developing a universal probe task to assess how information is propagated through each module of each decoder layer. We perform extensive experiments on three major translation datasets (WMT En-De, En-Fr, and En-Zh). Our analysis provides insight on when and where decoders leverage different sources. Based on these insights, we demonstrate that the residual feed-forward module in each Transformer decoder layer can be dropped with minimal loss of performance – a significant reduction in computation and number of parameters, and consequently a significant boost to both training and inference speed.

## introduction transformer models have advanced the state-ofthe-art on a variety of natural language processing (nlp) tasks, including machine translation @xcite , natural language inference @xcite , semantic role labeling @xcite , and language representation @xcite . however, so far not much is known about the internal properties and functionalities it learns to achieve its superior performance, which poses significant challenges for human understanding of the model and potentially designing better architectures. ## related work interpreting encoder representations previous studies generally focus on interpreting the encoder representations by evaluating how informative they are for various linguistic tasks @xcite , for both rnn models @xcite @xcite and transformer models @xcite @xcite . although they found that a certain amount of linguistic information is captured by encoder representations, it is still unclear how much encoded information is used by the decoder. our work bridges this gap by interpreting how the transformer decoder exploits the encoded information. interpreting encoder attention the encoderattention weights are generally employed to interpret the output predictions of nmt models. recently, @xcite showed that atten- ## conclusions in this paper, we interpreted nmt transformer decoder by assessing the evolution of both source and target information across layers and modules. to this end, we investigated the information functionalities of decoder components in the translation process. experimental results on three major datasets revealed several findings that help understand the behaviors of transformer decoder from different perspectives. we hope that our analysis and findings could inspire architectural changes for further improvements, such as 1) improving the word alignment of higher sems by incorporating external alignment signals; 2) exploring the stacking order of sem, tem and ifm sub-layers, which may provide a more effective way to transform information; 3) further pruning redundant sub-layers for efficiency. since our analysis approaches are not limited to the transformer model, it is also interesting to explore other architectures such as rnmt @xcite , convs2s @xcite , or on document-level nmt @xcite . in addition, our analysis methods can be applied to other sequence-to-sequence tasks such as summarization and grammar error correction, whose source and target sides are in the same language. we leave those tasks for future work.