The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? (2020)

Abstract: There is a recent surge of interest in using attention as explanation of model predictions, with mixed evidence on whether attention can be used as such. While attention conveniently gives us one weight per input token and is easily extracted, it is often unclear toward what goal it is used as explanation. We find that often that goal, whether explicitly stated or not, is to find out what input tokens are the most relevant to a prediction, and that the implied user for the explanation is a model developer. For this goal and user, we argue that input saliency methods are better suited, and that there are no compelling reasons to use attention, despite the coincidence that it provides a weight for each input. With this position paper, we hope to shift some of the recent focus on attention to saliency methods, and for authors to clearly state the goal and user for their explanations.

## introduction attention mechanisms @xcite have allowed for performance gains in many areas of nlp, including, inter alia, machine translation @xcite @xcite , natural language generation (e.g., @xcite , and natural language inference (e.g., @xcite . attention has not only allowed for better performance, it also provides a window into how a model is operating. for example, for machine translation, @xcite visualize what source tokens the target tokens are attending to, often aligning words that are translations of each other. whether the window that attention gives into how a model operates amounts to explanation has recently become subject to debate ( §2). while many papers published on the topic of explainable ai have been criticised for not defining explanations @xcite , the first key studies which spawned interest in attention as explanation @xcite @xcite do say that they are interested in whether attention weights faithfully represent the responsibility each input token has on a model prediction. that is, the narrow definition of explanation implied there is that it points at the most important input tokens for a prediction (arg max), accurately summarizing the reasoning process of the model @xcite . the above works have inspired some to find ways to make attention more faithful and/or plausible, by changing the nature of the hidden representations attention is computed over using special training objectives (e.g., @xcite . others have proposed replacing the attention mechanism with a latent alignment model @xcite . interestingly, the implied definition of explanation in the cited works, happens to coincide with what input saliency methods ( §3) are designed to produce @xcite @xcite . moreover, the user of that explanation is often implied to be a model developer, to which faithfulness is important. the elephant in the room is therefore: if the goal of using attention as explanation is to assign importance weights to the input tokens in a faithful manner, why should the attention mechanism be preferred over the multitude of existing input saliency methods designed to do exactly that? in this position paper, with that goal in mind, we argue that we should pay attention no heed ( §4). we propose that we reduce our focus on attention as explanation, and shift it to input saliency methods instead. however, we do emphasize that understanding the role of attention is still a valid research goal ( §5), and finally, we discuss a few approaches that go beyond saliency ( §6). ## the attention debate in this section we summarize the debate on whether attention is explanation. the debate mostly features simple bilstm text classifiers (see figure 1 ). unlike transformers @xcite , they only contain a single attention mechanism, which is typically mlp-based @xcite : when there is a single input text, there is no query, and q is either a trained parameter (like v, w h and w q ), or w q q is simply left out of eq. 1. ## saliency methods in this section we discuss various input saliency methods for nlp as alternatives to attention: gradient-based ( §3.1), propagation-based ( §3.2), and occlusion-based methods ( §3.3), following @xcite . we do not endorse any specific method foot_0 , but rather try to give an overview of methods and how they differ. we discuss methods that are applicable to any neural nlp model, allowing access to model internals, such as activations and gradients, as attention itself requires such access. we leave out more expensive methods that use a surrogate model, e.g., @xcite . ## saliency vs. attention we discussed the use of attention as explanation ( §2) and input saliency methods as alternatives ( §3). we will now argue why saliency methods should be preferred over attention for explanation. in many of the cited papers, whether implicitly or explicitly, the goal of the explanation is to reveal which input words are the most important ones for the final prediction. this is perhaps a consequence of attention computing one weight per input, so it is necessarily understood in terms of those inputs. the intended user for the explanation is often not stated, but typically that user is a model developer, and not a non-expert end user, for example. for model developers, faithfulness, the need for an explanation to accurately represent the reasoning of the model, is a key concern. on the other hand, plausibility is of lesser concern, because a model developer aims to understand and possibly improve the model, and that model does not necessarily align with human intuition (see @xcite , for a detailed discussion of the differences between faithfulness and plausibility). with this goal and user clearly stated, it is impossible to make an argument in favor of using attention as explanation. input saliency methods are addressing the goal head-on: they reveal why one particular model prediction was made in terms of how relevant each input word was to that prediction. moreover, input saliency methods typically take the entire computation path into account, all the way from the input word embeddings to the target output prediction value. attention weights do not: they reflect, at one point in the computation, how much the model attends to each input representation, but those representations might already have mixed in information from other inputs. ironically, attention-as-explanation is sometimes evaluated by comparing it against gradient-based measures, which again begs the question why we wouldn't use those measures in the first place. one might argue that attention, despite its flaws, is easily extracted and computationally efficient. however, it only takes one line in a framework like tensorflow to compute the gradient of the output w.r.t. the input word embeddings, so implementation difficulty is not a strong argument. in terms of efficiency, it is true that for attention only a forward pass is required, but many other methods discussed at most require a forward and then a backward pass, which is still extremely efficient. ## attention is not not interesting in this position paper we criticized the use of attention to assess input saliency for the benefit of the model developer. we emphasize that understanding the role of the attention mechanism is a perfectly justified research goal. for example, @xcite and @xcite analyze the role of attention heads in the transformer architecture and identify a few distinct functions they have, and @xcite train attention heads to perform dependency parsing, adding a linguistic bias. we also stress that if the definition of explanation is adjusted, for example if a different intended user and a different explanatory goal are articulated, attention may become a useful explanation for a certain application. for example, @xcite demonstrate that supervised attention helps humans accomplish a task faster than random or unsupervised attention, for a user and goal that are very different from those implied in §2. beyond saliency. while we have argued that saliency methods are a good fit for our goal, there are other goals for which different methods can be a better fit. for example, counterfactual analysis might lead to insights, aided by visualization tools @xcite @xcite . the diffmask method of decao et al. (2020) adds another dimension: it not only reveals in what layer a model knows what inputs are important, but also where important information is stored as it flows through the layers of the model. other examples are models that rationalize their predictions @xcite , which can guarantee faithful explanations, although they might be sensitive to so-called trojans @xcite . ## conclusion we summarized the debate on whether attention is explanation, and observed that the goal for explanation is often to determine what inputs are the most relevant to the prediction. the user for that explanation often goes unstated, but is typically assumed to be a model developer. with this goal and user clearly stated, we argued that input saliency methods-of which we discussed a few-are better suited than attention. we hope, at least for the goal and user that we identified, that the focus shifts from attention to input saliency methods, and perhaps to entirely different methods, goals, and users.