Training with Adversaries to Improve Faithfulness of Attention in Neural Machine Translation (2020)

Abstract: Can we trust that the attention heatmaps produced by a neural machine translation (NMT) model reflect its true internal reasoning? We isolate and examine in detail the notion of faithfulness in NMT models. We provide a measure of faithfulness for NMT based on a variety of stress tests where model parameters are perturbed and measuring faithfulness based on how often the model output changes. We show that our proposed faithfulness measure for NMT models can be improved using a novel differentiable objective that rewards faithful behaviour by the model through probability divergence. Our experimental results on multiple language pairs show that our objective function is effective in increasing faithfulness and can lead to a useful analysis of NMT model behaviour and more trustworthy attention heatmaps. Our proposed objective improves faithfulness without reducing the translation quality and it also seems to have a useful regularization effect on the NMT model and can even improve translation quality in some cases.

## introduction can we trust our neural models? this question has led to a wide variety of contemporary nlp research focusing on (a) different axes of interpretability including plausibility (or interchangeably human-interpretability) @xcite and faithfulness @xcite , (b) interpretation of the neural model components @xcite @xcite , (c) explaining the decisions made by neural models to humans (using explanations, highlights, rationales, etc.) @xcite @xcite @xcite , and (d) evaluating different explanation methods from different perspectives in the left attention heatmap, the attention is on the word sto while the decoder generates century. however, in the right heatmap, sto is not attended to at all but century is still produced as the output. this is an example of unfaithful behavior. yellow words are not attended. @xcite @xcite @xcite @xcite . @xcite emphasize distinguishing faithfulness from human-interpretability in interpretability research by providing several clarifications about the terminology used by researchers. they describe the following conditions on the evaluation of how well a research project tackles the notion of faithfulness: (1) be explicit: provide a measurable evaluation of faithfulness, (2) human judgements are not relevant because we are interested in model internals, (3) do not match against gold labels (e.g. aer) because faithfulness of both correct and incorrect decisions made by the model are equally important, (4) no model is "inherently" faithful. we need to measure faithfulness not as a binary aspect of a model but rather as a gray-scale measure. aligned with these criteria, we study faithfulness of attention in nmt, the extent to which it can reflect the true internal reasoning behind a prediction (figure 1 ). we make the following contributions: we chose to study the impact of faithfulness in nmt because it is under-studied in terms of interpretability. most previous work has focused on document or sentence-based classification tasks where attention models are not as directly useful as in nmt models. attention is also more challenging in terms of faithfulness in the context of nmt models due to the substantial impact of the decoder component. foot_0 ## faithfulness in nmt models intuitively, a faithful explanation should reflect the true internal reasoning of the model. although there is no formal definition for faithfulness, a common approach in the community is to design stress tests to perturb the model parameters chosen in such a way that the model's decision should change if the model is faithful @xcite . a common stress test is the erasure test in which the most-relevant part of the input is removed @xcite . in the context of nmt, at decoding time step t the attention component assigns attention weights α t , attending to the source word at position m @xmath0. these weights are often implicitly or explicitly regarded as an interpretation for the model's prediction at the time step t @xcite @xcite @xcite @xcite . the erasure stress test for evaluating faithfulness offered by α t is done by setting α t [m t ] to zero and observing whether or not the output changes. it is worth noting that erasure is only one of the possible stress tests for evaluating faithfulness. passing more stress tests implies a more faithful model as it is resilient to more attacks or stress tests of its faith-fulness. in this paper we consider three intuitive stress test cases: the attention weights are randomly permuted until a change in the model output is observed. we ensure that m t , the most important token according to attention, is always changed. we many prior studies of attention @xcite have used a binary measure: either attention is faithful or it is not. these studies typically are about whether attention has the potential to be useful in terms of accuracy and faithful in terms of model behaviour. in many cases, especially in the case of nmt models, attention is clearly useful and by and large it must be faithful. the question is can we measure the faithfulness and improve faithfulness. it is more natural to have a gray-scale notion of faithfulness for evaluation @xcite . following this reasoning, we define f (m ) as faithfulness of attention heatmaps in model m as the following equation: ## approach the conventional objective function in a sequenceto-sequence task is a cross-entropy loss f acc : attention layer context vector attn weights stress tests zom uni perm figure 2: we generate adversaries to the attention weights using various stress tests uniform, zerooutmax, and random-permute. when adversarial attention weights are used, in a faithful model we expect the probability of the original output (ŷ) to drop significantly. we use this criteria to define a faithfulness objective function. faithfulness objective in an effort to develop a model that is right for right reason, ross et al. f f aith is an additional component that rewards the model for having more faithful attention. the parameter λ f aith regulates the trade-off between between faithfulness and accuracy objectives. ## experimental setup we use the czech-english (cs-en) dataset from iwslt2016and the german-english (de-en) dataset from iwslt2014.we used moses @xcite to tokenize the dataset. we use open-nmt @xcite as our translation framework. we employ a 2 layer lstm-based encoderdecoder @xcite model with global attention @xcite . we use adam (kingma and ba, 2014) for training our models and we set the learning rate to 0.001. models are trained until convergence. the baseline model is trained using eqn. (2) and we call it f baseline . we refer to the objective as f all when λ zom , λ uni , and λ perm are set to 0.5, 0.375, and 0.125 respectively. λ f aith is set to 1. ## related work while several studies have focused on understanding the semantic notions captured by attention @xcite @xcite , evaluating attention as an interpretability approach has garnered a lot of interest. from the faithfulness perspective, @xcite show that for instances in a data set there can be adversarial attention heatmaps that do not change the output of the text classifier. in other words, adversarial attention leads to no decision flip in each instance. they use this to claim that attention heatmaps are not to be trusted, or unfaithful. @xcite argue against per-instance modifications at test time for two reasons: 1) in classification tasks attention may not be useful so perturbing attention is misleading. this is not true for nmt since attention is very useful in nmt. 2) they train an adversarial attention model (e.g. uniform attention) chosen to produce attention weights distant from the original attention weights while at the same time trying to minimize classification error. they show that such adversarial attention models are not as accurate as models with attention. in our work we acknowledge that attention is useful and faithful to some extent and we aim to improve faithfulness of nmt models. while most of these works provide evidence that attention weights are not always faithful, @xcite confirm similar observations on the unfaithful nature of attention in the context of nmt models. @xcite is one of the few papers examining attention models in nmt. however, they are focused on the task of analyzing attention weights from fidelity perspective which is different from faithfulness. while prior works have mostly failed to explicitly distinguish faithfulness from plausibility in their arguments, @xcite focus on formalizing faithfulness and addressing evaluation of faithfulness separately from plausibility respectively. ## conclusion in this paper, we proposed a method for quantifying faithfulness of nmt models. to optimize faithfulness we have defined a novel objective function that rewards faithful behavior through probability divergence. unlike previous work, our method does not use prior knowledge or extraneous data. we also show that the additional constraint in the training objective for nmt does not harm translation quality and in some cases we see some better translations presumably due to the regularization effect of our faithfulness objective.