Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer (2022)

Abstract: In Neural Machine Translation (NMT), each token prediction is conditioned on the source sentence and the target prefix (what has been previously translated at a decoding step). However, previous work on interpretability in NMT has mainly focused solely on source sentence tokens’ attributions. Therefore, we lack a full understanding of the influences of every input token (source sentence and target prefix) in the model predictions. In this work, we propose an interpretability method that tracks input tokens’ attributions for both contexts. Our method, which can be extended to any encoder-decoder Transformer-based model, allows us to better comprehend the inner workings of current NMT models. We apply the proposed method to both bilingual and multilingual Transformers and present insights into their behaviour.

## introduction transformers @xcite have become the state-of-the-art architecture for natural language processing (nlp) tasks @xcite @xcite . with its success, the nlp community has experienced an urge to understand the decision process of the model predictions @xcite . in neural machine translation (nmt), attempts to interpret transformer-based predictions have mainly focused on analyzing the attention mechanism @xcite . a large number of works in this line have investigated the capabilities of the cross-attention to perform source-target alignment @xcite @xcite , compared with human annotations. gradient-based @xcite and occlusion-based methods @xcite have also been evaluated against human word alignments. the former computes gradients with respect to the input token embeddings to measure how much a change in the input changes the output, the latter generates input attributions by measuring the change in the predicted probability after deleting specific tokens. however, there is a tension between finding a faithful explanation and observing human-like alignments, since one does not imply the other @xcite . the decoding process of nmt systems consists of generating tokens in the target vocabulary based on the information provided by the source sequence and the previously generated tokens (target prefix). however, most of the work on interpretability of nmt models only analyses source tokens. recently, @xcite proposed using layer relevance propagation (lrp) @xcite to analyze the source and target contributions to the model prediction, and later analyzed its behaviour during training @xcite . nonetheless, they apply their method to obtain global explanations, as an average over the entire dataset, not to get input attributions of a single prediction. gradient-based methods have also been extended to the target prefix @xcite ), although they do not quantify the relative contribution of source and target inputs. concurrently, encoder-based transformers, such as bert @xcite and roberta @xcite , have been analysed with attention rollout @xcite , which models the information flow in the model with a directed acyclic graph, where nodes are token representations and edges, attention weights. in the computer vision literature, @xcite combined this method with gradient information. recently, @xcite have presented alti (aggregation of layer-wise tokens attributions), which applies the attention rollout method by substituting attention weights with refined token-to-token interactions. in this work, we present the first application of a rollout-based method to sequence to sequence transformers. our key contributions are 1 : ## background in this section, we provide the background to understand our proposed method by briefly explaining the encoder-decoder transformer-based model in the context of nmt @xcite and the aggregation of layer-wise token-to-token interactions (alti) method @xcite . ## alti for the encoder-decoder transformer (alti+) the attention rollout and alti methods work for encoder-based transformers. however, in the encoder-decoder transformer, the cross-attention hinders its integration. in this section, we present alti+, which is the adaptation of alti method to the encoder-decoder transformer. ## experimental setup we analyze input token attributions in both bilingual and multilingual machine translation models. for the bilingual setting, we train a 6-layer transformer model for the german-english (de-en) translation task. we use europarl v7 corpus foot_5 and follow @xcite and @xcite data setup foot_6 . we use byte-pair encoding (bpe) @xcite with 10k merge operations. for the multilingual model, we use m2m transformer @xcite , a many-to-many multilingual translation model that can translate directly between any pair of 100 languages. we use fairseq @xcite implementations, and the provided checkpoint for the m2m model (418m). we perform the quantitative analysis in 1000 sentences of the test set of iwslt'14 german-english dataset. for the analysis in §5.5 we use flores-101 @xcite devtest split. ## analysis in this section, we perform a set of experiments to measure the quality of the obtained contributions, and unveil different aspects of bilingual and multilingual nmt models. ## conclusions we propose alti+, an interpretability method for the encoder-decoder transformer that provides token influences to the model predictions for the two input contexts: source sentence and target prefix. by applying alti+ to a bilingual and a multilingual nmt model we are able to discover insights into the behaviour of these black-box models. unlike previous methods, we can now observe dependencies between tokens in the predicted sentence, and quantify the total contribution of each of the contexts. this allows a deeper exploration of current nmt models. our findings include: the role of the source eos (</@xmath0, the absence of source contribution when producing hallucinations, and the lack of source contributions when translating from english to a low-resource language. alti+ overcomes the limitations of previous interpretability methods in nmt, and we believe it can help researchers and practitioners to better understand any encoder-decoder transformer model.