Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions (2021)

Abstract: This work proposes an extensive analysis of the Transformer architecture in the Neural Machine Translation (NMT) setting. Focusing on the encoder-decoder attention mechanism, we prove that attention weights systematically make alignment errors by relying mainly on uninformative tokens from the source sequence. However, we observe that NMT models assign attention to these tokens to regulate the contribution in the prediction of the two contexts, the source and the prefix of the target sequence. We provide evidence about the influence of wrong alignments on the model behavior, demonstrating that the encoder-decoder attention mechanism is well suited as an interpretability method for NMT. Finally, based on our analysis, we propose methods that largely reduce the word alignment error rate compared to standard induced alignments from attention weights.

## introduction recently, transformer-based models @xcite have allowed huge improvements in performance across multiple nlp tasks. the inclusion of this architecture has led the field of nlp to investigate the inner workings of this architecture in several tasks. one of its core components, the attention mechanism, which provides a distribution of scores over the input tokens, has been often presented as showing the relative importance of the inputs. some works have criticized the use of attention weights as model explanations @xcite @xcite , demonstrating that attention weights distributions can be modified without affecting the final prediction. however, these studies have mainly analyzed encoder-only or decoder-only architectures like bert @xcite or @xcite , which are based on self-attention mechanisms. nonetheless, nmt models use the encoderdecoder transformer architecture, which adds the encoder-decoder attention mechanism, in charge of distributing the information flow from the encoder representations of the source input tokens into the decoder. @xcite analyze the effect of pruning different attention heads in a transformer nmt model and conclude that the encoderdecoder attention mechanism is the most critical one. @xcite show that encoder selfattention weights can be interchanged by predefined non-learnable patterns without hindering the translation performance. these results provide evidence about the relevance of the encoder-decoder attention mechanism on nmt, which we believe needs further investigation. in this work we analyze the encoder-decoder attention weights and shed light on their impact on the decoder representations and final predictions, showing how alignment errors can also give information about the model's decision-making process. research in nmt interpretability has mainly focused on understanding source words importance when predicting a target word. the word alignment task @xcite has served to compare explanation methods against human-annotated source-target word alignments. encoder-decoder attention weights have been used to provide sourcetarget word alignments @xcite , but its low performance has made researchers sceptical about its use as an interpretable method @xcite . an important issue when relying on word alignment task is that it ignores the words that are predicted based on the target prefix, i.e what the model has previously translated. an extreme example of the impact of the target prefix on the prediction occurs during 'hallucinations' @xcite @xcite . although some studies have analyzed the relative contribution of the target prefix context in a model's prediction @xcite , the way nmt models decide in which proportion to use both sequences remains unexplored. in section 3, we propose a simple method to measure the relative contribution of the source and the target prefix by perturbing input embeddings, we also extend the gradient-based method towards the target prefix token embeddings to obtain saliency scores from the prefix words. both methods serve us in section 5 to understand the attention weights generated in the encoder-decoder modules and their relationship with the model predictions. lastly, in section 6, we propose two methods to improve the alignment error extracted from the model in accordance with our results analysis. ## background in this section, we briefly introduce the existing methodologies that we use in our work: the transformer and the methods used to induce alignment. in terms of the model's input and output, we consider x j and y i representing the embeddings of each token from the source and target prefix respectively. so, we can write the conditional probability modeled by the network at each time step as: ## proposed methods for analysis in this section, we introduce two simple methods for measuring the contributions of each source sequence to a model prediction and extend the gradient-based analysis to understand dependency relationships between target prefix words. ## experimental setup as follows, we detail the model and datasets used in our experiments. we decide to choose this experimental framework to compare and further explain previous works @xcite @xcite . we follow the same procedure as these past works, we train the transformer model for the german-english translation task. specifically, we use europarl v7 corpus foot_0 which consists on 1.9m sentence pairs. we use the gold alignment dataset foot_1 (vilar et al., 2006) which contains 508 sentence pairs. the transformer used in this work foot_2 is implemented in fairseq (ott et al., 2019) and contains 6 layers with 4 attention heads each. we apply byte pair encoding (bpe) (sennrich et al., 2016) with 10k merging operations. as (ding et al., 2019) and (kobayashi et al., 2020 ## analysis in this section, and for the sake of clarity, we convey our analysis on a single example while quantifying how our findings generalize to the entire test set. we use the following source example: ## methods to improve alignment as shown in the previous analysis, alignments from attention weight matrices reveal errors mainly due to the existence of the skip source attention operation. in this section we propose two methods to get more clear alignments. ## conclusion in this paper, we have studied the use of attention weights as an explanatory method for the transformer in nmt. we have proposed analysis methods that measure the relative contribution of the source and the target prefix sequences. then, we have demonstrated that the alignment bias towards finalizing tokens, which is the most common alignment error, is used by the model to avoid source information flowing through the decoder. in these cases, the predicted output relies on prefix dependencies, which are identifiable by extending the gradient-based analysis to extract saliency scores. furthermore, we have proposed two methods to improve the extraction of alignments from attention weights. as future work, we plan to extend our study to more languages pairs, as well as to the multilingual nmt setting.