Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation (2021)

Abstract: In Neural Machine Translation (and, more generally, conditional language modeling), the generation of a target token is influenced by two types of context: the source and the prefix of the target sequence. While many attempts to understand the internal workings of NMT models have been made, none of them explicitly evaluates relative source and target contributions to a generation decision. We argue that this relative contribution can be evaluated by adopting a variant of Layerwise Relevance Propagation (LRP). Its underlying ‘conservation principle’ makes relevance propagation unique: differently from other methods, it evaluates not an abstract quantity reflecting token importance, but the proportion of each token’s influence. We extend LRP to the Transformer and conduct an analysis of NMT models which explicitly evaluates the source and target relative contributions to the generation process. We analyze changes in these contributions when conditioning on different types of prefixes, when varying the training objective or the amount of training data, and during the training process. We find that models trained with more data tend to rely on source information more and to have more sharp token contributions; the training process is non-monotonic with several stages of different nature.

## introduction with the success of neural approaches to natural language processing, analysis of nlp models has become an important and active topic of research. in nmt, approaches to analysis include probing for linguistic structure @xcite , evaluating via contrastive translation pairs @xcite @xcite , inspecting model components, such as attention @xcite @xcite @xcite or neurons @xcite , among others. unfortunately, although a lot of work on model analysis has been done, a question of how the nmt predictions are formed remains largely open. namely, the generation of a target token is defined by two types of context, source and target, but there is no method which explicitly evaluates the relative contribution of source and target to a given prediction. the ability to measure this relative contribution is important for model understanding since previous work showed that nmt models often fail to effectively control information flow from source and target contexts. for example, adding context gates to dynamically control the influence of source and target leads to improvement for both rnn @xcite and transfomer @xcite models. a more popular example is a model's tendency to generate hallucinations (fluent but inadequate translations); it is usually attributed to the inappropriately strong influence of target context. several works observed that, when hallucinating, a model fails to properly use source: it produces a deficient attention matrix, where almost all the probability mass is concentrated on uninformative source tokens (eos and punctuation) @xcite . we argue that a natural way to estimate how the source and target contexts contribute to generation is to apply layerwise relevance propagation (lrp) @xcite to nmt models. lrp redistributes the information used for a prediction between all input elements keeping the total contribution constant. this 'conservation principle' makes relevance propagation unique: differently from other methods estimating influence of individual tokens @xcite @xcite , lrp evaluates not an abstract quantity reflecting a token importance, but the proportion of each token's influence. we extend one of the lrp variants to the transformer and conduct the first analysis of nmt models which explicitly evaluates the source and target relative contributions to the generation process. we analyze changes in these contributions when conditioning on different types of prefixes (reference, generated by a model or random translations), when varying training objective or the amount of training data, and during the training process. we show that models suffering from exposure bias are more prone to over-relying on target history (and hence to hallucinating) than the ones where the exposure bias is mitigated. when comparing models trained with different amount of data, we find that extra training data teaches a model to rely on source information more heavily and to be more confident in the choice of important tokens. when analyzing the training process, we find that changes in training are non-monotonic and form several distinct stages (e.g., stages changing direction from decreasing influence of source to increasing). our key contributions are as follows: ## layer-wise relevance propagation layer-wise relevance propagation is a framework which decomposes the prediction of a deep neural network computed over an instance, e.g. an image or sentence, into relevance scores for single input dimensions of the sample such as subpixels of an image or neurons of input token embeddings. the original lrp version was developed for computer vision models @xcite and is not directly applicable to the transformer (e.g., to the attention layers). in this section, we explain the general idea behind lrp, specify which of the existing lrp variants we use, and show how to extend lrp to the nmt transformer model. foot_1 ## experimental setting model. we follow the setup of transformer base model @xcite with the standard training setting. more details on hyperparameters and the optimizer can be found in the appendix. ## exposure bias and source contributions the results in the previous section agree with some observations made in previous work studying selfrecovery and hallucinations. in this section, we illustrate more explicitly how our methodology can be used to shed light on the effects of exposure bias and training objectives. @xcite empirically link the hallucination mode to exposure bias @xcite , i.e. the mismatch between the gold history seen at training time, and the (potentially erroneous) model-generated prefixes at test time. the authors hypothesize that exposure bias leads to an over-reliance on target history, and show that minimum risk training (mrt), which does not suffer from exposure bias, reduces hallucinations. however, they did not directly measure this overreliance on target history. our method is able to directly test whether there is indeed an over-reliance on the target history with mle-trained models, and more robust inclusion of source context with mrt. we also consider a simpler heuristic, word dropout, which we hypothesize to have a similar effect. (x,y) ỹ∈y(x) p (ỹ|x, θ)∆(ỹ, y), ## data amount in this section, we show how the results from section 4 change when increasing the amount of training data. the observed patterns are the same when evaluating on datasets with reference translations or the ones generated by the corresponding model (in each case, all sentences in the evaluation dataset have the same length). in the main text, we show figures for references. more @xmath0. figure 6(a) shows the source contribution at each generation step. we can see that, generally, models trained with more data rely on source more heavily. surprisingly, this increase is not spread evenly across positions: at approximately 80% of the target length, models trained with more data use source more, but at the last positions, they switch to more actively using the prefix. more @xmath1. figure 6(b) shows that at each generation step, entropy of source contributions decreases with more data. this means that with more training data, the model becomes more confident in the choice of important tokens. in the appendix, we show that this is also the case for target contributions. ## training stages now we turn to analyzing the training process of an nmt model. specifically, we look at the changes in how the predictions are formed: changes in the amount of source/target contributions and in the entropy of these contributions. our findings are summarized in figure 7 . in the following, we explain them in more detail. in section 7.1, we draw connections between our training stages (shown in figure 7 ) and the ones found in previous work focused on validating the lottery ticket hypothesis. contributions converge early. first, we evaluate how fast the contributions converge, i.e., how quickly a model understands which tokens are the most important for prediction. for (a) (b) (c) (d) figure 8: training process: (a) convergence of contributions, (b) source contribution, (c-d) entropy of source and target contributions. the model trained on 1m subsample of wmt14 en-fr dataset. the results are averaged over target positions and evaluation examples. changes in training are not monotonic. figures 8(b-d) show how the amount of source contribution and the entropy of source and target contributions change in training. we see that all three figures have the same distinct stages (shown with vertical lines). first, source influence decreases, and both source and target contributions become more focused. in this stage, most of the change happens (figure 8(a) ). in the second stage, the model also undergoes substantial change, but all processes change their direction: source influence increases and the model learns to rely on broader context (entropy is increasing). finally, in the third stage, the direction of changes remains the same, but very little is going on -the model slowly converges. these three stages correspond to the first three stages shown in figure 7 ; at this point, the model trained on 1m sentence pairs converges. with more data (e.g., 20m sentence pairs), we further observed the next stage (the last one in figure 7 ), where the entropy of both source and target contributions is decreasing again. however, this last stage is much slower than the third, and the final state does not differ much from the end of the third stage. early positions change more. figures 9(a-b ) show how source contributions and their entropy changes for each target position. we see that earlier positions are the ones that change most actively: at these positions, we see the largest decrease at the first stage and the largest following increase at the subsequent stages. if we look at how accuracy for each position changes in training (figure 10 ), we see that at the end of the first stage, early tokens have the highest accuracy. foot_6 this is not surprising: one could expect early positions to train faster because they are observed more frequently in training. previously such intuition motivated the usage of sentence length as one of the criteria for curriculum learning (e.g., @xcite ). ## additional related work to estimate the influence of source to an nmt prediction, @xcite trained an nmt model with an auxiliary second decoder where the encoder context vector was masked. then the source influence was measured as the kl divergence between predictions of the two decoders. however, the ability of an auxiliary decoder to generate similar distribution is not equivalent to the main model not using source. more recently, as a measure of individual token importance, @xcite used integrated gradients @xcite . in machine translation, lrp was previously used for visualization @xcite and to find the most important attention heads in the transformer's encoder @xcite . similar to our work, @xcite evaluated lrp on average over a dataset (and not for a single prediction) to extract patterns in model behaviour. both works used the more popular ε-lrp, while for our analysis, the αβ-lrp was more suitable (section 2). for language modeling, calvillo and crocker (2018) use lrp to evaluate relevance of neurons in rnns for a small synthetic setting. ## conclusions we show how to use lrp to evaluate the relative contributions of source and target to nmt predictions. we illustrate the potential of this approach by analyzing changes in these contributions when conditioning on different types of prefixes (references, model predictions or random translations), when varying training objectives or the amount of training data, and during the training process. some of our findings are: (1) models trained with more data rely on source information more and have more sharp token contributions; (2) the training process is non-monotonic with several distinct stages. these stages agree with the ones found in previous work focused on validating the lottery ticket hypothesis, which suggests future investigation of this connection. additionally, we show that models suffering from exposure bias are more prone to over-relying on target history (and hence to hallucinating) than the ones where the exposure bias is mitigated. in future work, our methodology can be used to measure the effects of different and novel training regimes on the balance of source and target contributions.