Probing Word Translations in the Transformer and Trading Decoder for Encoder Layers (2021)

Abstract: Due to its effectiveness and performance, the Transformer translation model has attracted wide attention, most recently in terms of probing-based approaches. Previous work focuses on using or probing source linguistic features in the encoder. To date, the way word translation evolves in Transformer layers has not yet been investigated. Naively, one might assume that encoder layers capture source information while decoder layers translate. In this work, we show that this is not quite the case: translation already happens progressively in encoder layers and even in the input embeddings. More surprisingly, we find that some of the lower decoder layers do not actually do that much decoding. We show all of this in terms of a probing approach where we project representations of the layer analyzed to the final trained and frozen classifier level of the Transformer decoder to measure word translation accuracy. Our findings motivate and explain a Transformer configuration change: if translation already happens in the encoder layers, perhaps we can increase the number of encoder layers, while decreasing the number of decoder layers, boosting decoding speed, without loss in translation quality? Our experiments show that this is indeed the case: we can increase speed by up to a factor 2.3 with small gains in translation quality, while an 18-4 deep encoder configuration boosts translation quality by +1.42 BLEU (En-De) at a speed-up of 1.4.

## introduction neural machine translation (nmt) has achieved great success in the last few years. the popular transformer @xcite model, which outperforms previous rnn/cnn based translation models @xcite , is based on multi-layer self-attention networks and can be parallelized effectively. recently, a wide range of studies related to the transformer have been conducted. for example, @xcite perform a fine-grained analysis of how various source-side morphological features are captured at different levels of an nmt encoder. surprisingly, they do not find any correlation between the accuracy of source morphology encoding and translation quality. morphological features are only captured in context and only to the extent that they are directly transferable to target words. @xcite study how information flows across transformer layers and find that representations differ significantly depending on the objectives (machine translation, standard left-toright language models and masked language modeling). @xcite find that encoder hidden states outperform word embeddings significantly in word sense disambiguation. however, to the best of our knowledge, to date there is no study about how the transformer translation model transforms individual source tokens into corresponding target tokens (i.e., word translations), and specifically, which role each transformer layer plays in word translation, and at which layer a word is translated. to investigate the roles of transformer layers in translation, in this paper, we adopt probing approaches @xcite @xcite and propose to measure the word translation accuracy of output representations of individual transformer layers by probing how capable they are at translating words. probing uses linear classifiers, referred to as "probes", where a probe can only use the hidden units of a given intermediate layer as discriminating features. moreover, these probes cannot affect the training phase of a model, and they are generally added after training @xcite . in addition to analyzing the role of each encoder/decoder layer, we also analyze the contribution of the source context and the decoding history in translation by testing the effects of the masked self-attention sub-layer and the cross-attention sub-layer in decoder layers. we present empirical results for how word translation is performed in each encoder/decoder layer, and how the alignment modeling (cross-attention sub-layers) and language modeling (masked selfattention sub-layers) contribute to the performance in each decoder layer. our analysis demonstrates how word translation evolves across encoder/decoder layers and provides insights into the impact of the source "encoding" and the decoding history on the translation of target tokens. it reveals the existence of target translations in encoder states (and even source word embeddings) and the translation performed by encoder layers. based on our findings, we show that the proper use of more encoder layers with fewer decoder layers can significantly boost decoding speed without harming quality. recently, @xcite independently and similar to our encoder-decoder layer trading approach, compare the performance and speed of a 12-layer encoder 1-layer decoder with non-autoregressive translation (nat) approaches, and show that a one-layer autoregressive decoder can yield state-of-the-art accuracy with comparable latency to strong non-autoregressive models. our analysis explains why using a deep encoder with a shallow decoder is feasible, and we show that some encoder-decoder depth configurations deliver both increased speed and increased translation quality. ## probing layer-wise word translation to analyze word translation accuracy of the transformer, we first freeze a trained transformer model so its behavior is consistent in how it performs in translation during our analysis. we then extract output representations of the particular layer analyzed, apply a linear projection layer to extract features related to translation and feed the projected representations to the frozen decoder classifier of the trained transformer. our approach is minimally invasive in that only the linear projection layer and the weights of the alignment matrix a responsible for combining frozen cross-attention alignment matrices from the decoder are trained and updated on the training set, with the original transformer being frozen. thus the projection layer will only transform between vector spaces without generating new features for the word translation, and the alignment matrix a will only combine frozen crossattention alignment matrices. a high-level illustration of our analysis approach for encoder/decoder layers is shown in figure 1 . ## conclusion we propose approaches for the analysis of word translation accuracy of transformer layers to investigate how translation is performed. to measure word translation accuracy, our approach trains a linear projection layer that bridges representations from the frozen pre-trained analyzed layer and the frozen pre-trained classifier. while analyzing encoder layers, our approach additionally learns a weight vector to merge multiple attention matrices into one, and transforms the source "encoding" to the target shape by multiplying the merged alignment matrix. both the linear projection layer and the weight vector are trained on the frozen transformer. this is minimally invasive, and training the new parameters does not account for the findings reported. for the analysis of decoder layers, we additionally analyze the effects of the source context and the decoding history in word prediction through bypassing the corresponding cross-and self-attention sub-layers. our findings motivate and explain the benefits of trading decoder for encoder layers in our approach and that of @xcite . our analysis is the first to reveal the existence of target translations performed by encoder layers (including the source embedding layer). we show that increasing encoder depth while removing decoder layers can lead to significant bleu improvements while boosting the decoding speed.