Measuring and Improving Faithfulness of Attention in Neural Machine Translation (2021)

Abstract: While the attention heatmaps produced by neural machine translation (NMT) models seem insightful, there is little evidence that they reflect a model’s true internal reasoning. We provide a measure of faithfulness for NMT based on a variety of stress tests where attention weights which are crucial for prediction are perturbed and the model should alter its predictions if the learned weights are a faithful explanation of the predictions. We show that our proposed faithfulness measure for NMT models can be improved using a novel differentiable objective that rewards faithful behaviour by the model through probability divergence. Our experimental results on multiple language pairs show that our objective function is effective in increasing faithfulness and can lead to a useful analysis of NMT model behaviour and more trustworthy attention heatmaps. Our proposed objective improves faithfulness without reducing the translation quality and has a useful regularization effect on the NMT model and can even improve translation quality in some cases.

## introduction how trustworthy are our neural models? this question has led to a wide variety of contemporary nlp research focusing on (a) different axes of interpretability including plausibility (or interchangeably human-interpretability) @xcite and faithfulness @xcite , (b) interpretation of the neural model components @xcite @xcite , (c) explaining the decisions made by neural models to humans (using explanations, highlights, rationales, etc.) @xcite @xcite @xcite , and (d) evaluating different explanation methods from different perspectives je to moorův zákon za posledních sto let je to moorův zákon za posledních sto let it's moore's law for the last century it's moore's law for the last century 0.00 1.00 attention weights figure 1 : an example of unfaithful attention weights produced during a cs-en translation. note in the left attention heatmap, the attention is on the word sto while the decoder generates century. however, in the right heatmap, sto is not attended to at all but century is still produced as the output. this shows unfaithful behavior. @xcite @xcite @xcite @xcite . all of these approaches make nlp neural models more trustworthy. in this work, we focus on faithfulness which intuitively provides the extent to which an explanation accurately represents the true reasoning behind a prediction. it is particularly important for nlp practitioners who wish to debug their neural models and improve them. faults of a neural model cannot be identified if the neural model does not provide a faithful and trustworthy description of what it is doing. however, the formal definition of faithfulness and the proper approach to its evaluation are still contended in the literature. @xcite emphasize distinguishing faithfulness from human-interpretability in interpretability research by providing several clarifications about the terminology used by researchers. they describe the following conditions on the evaluation of how well a research project tackles the notion of faithfulness: aligned with these criteria, we study faithfulness of nlp neural models, specifically nmt models. we provide a faithfulness measure that is computed based on a variety of stress tests where attention weights that are crucial for prediction are perturbed. we expect from a faithful model to change its prediction under such tests (figure 1 ). we quantify faithfulness based on how often the model outputs changed. the proposed metric is defined based on discrete changes in the output. it is not differentiable and cannot be simply included in the loss function of nmt to be optimized. we propose a novel differentiable objective based on probability divergence and study its effect on the discrete faithfulness measure. our findings show that our objective is effective in increasing faithfulness and can lead to a useful analysis of nmt model behaviour and more trustworthy attention heatmaps. we assert that faithfulness is a good property to have in a model whether or not it will be useful for downstream interpretation. a model that is faithful can be trusted better as a component in a larger end-to-end neural model. contributions we seek to improve faithfulness of nmt models. to this end, we make the following contributions in this work: ## faithfulness in nmt models intuitively, a faithful explanation should reflect the true internal reasoning of the model. although there is no formal definition for faithfulness, a common approach in the community is to design stress tests to perturb the model parameters chosen in such a way that the model's decision should change if the model is faithful @xcite . a common stress test is the erasure test in which the most-relevant part of the input is removed @xcite . in the context of nmt, at decoding time step t the attention component assigns attention weights α t , attending to the source word at position m @xmath0. these weights are often implicitly or explicitly regarded as an interpretation for the model's prediction at the time step t @xcite @xcite @xcite @xcite . the erasure stress test for evaluating faithfulness offered by α t is done by setting α t [m t ] to zero and observing whether or not the output changes. it is worth noting that erasure is only one of the possible stress tests for evaluating faithfulness. passing more stress tests implies a more faithful model as it is properly reacting to more adversaries by changing its decision. in this paper we consider three intuitive stress test cases: zerooutmax: @xcite : here we remove attention from the most important token according to the attention weights by setting α t [m t ] = 0. uniform: @xcite : in this stress test all attention weights are set to be equal, α @xmath1, where m is the length of the source sentence. this is to confuse the model about which part of the input is the most important one. randompermute: (jain and wallace, 2019): in this stress test we randomly permute attention weights several times until a change in the model output is observed. we ensure that m t , the most important token according to attention, is always changed. we set α @xmath2: either attention is faithful or it is not. these studies typically are about whether attention has the potential to be useful in terms of accuracy and faithful in terms of model behaviour. in many cases, especially in the case of nmt models, attention is clearly useful and by and large it must be faithful. the question is can we measure the faithfulness and improve faithfulness. it is more natural to have a gray-scale notion of faithfulness for evaluation @xcite . following this reasoning, we define f (m ) as faithfulness of attention heatmaps in model m as the following equation: ## approach the conventional objective function in a sequenceto-sequence task is a cross-entropy loss f acc : attention layer context vector attn weights stress tests zom uni perm figure 2: using zerooutmax, uniform, and random-permute stress tests, we generate adversaries to the attention weights. when adversarial attention weights are used, in a faithful model we expect the probability of the original output (ŷ) to drop significantly. we use this criterion to define a faithfulness objective function. ## conclusion we proposed a method for quantifying faithfulness of nmt models. to optimize faithfulness we have defined a novel objective function that rewards faithful behavior through probability divergence. we also show that the additional constraint in the training objective for nmt does not harm transla-tion quality and in some cases we see some better translations presumably due to the regularization effect of our faithfulness objective.