Is Attention Explanation? An Introduction to the Debate (2022)

Abstract: The performance of deep learning models in NLP and other fields of machine learning has led to a rise in their popularity, and so the need for explanations of these models becomes paramount. Attention has been seen as a solution to increase performance, while providing some explanations. However, a debate has started to cast doubt on the explanatory power of attention in neural networks. Although the debate has created a vast literature thanks to contributions from various areas, the lack of communication is becoming more and more tangible. In this paper, we provide a clear overview of the insights on the debate by critically confronting works from these different areas. This holistic vision can be of great interest for future works in all the communities concerned by this debate. We sum up the main challenges spotted in these areas, and we conclude by discussing the most promising future avenues on attention as an explanation.

## introduction attention mechanisms have been widely used in various tasks of natural language processing (nlp) as well as in other fields of machine learning (e.g., computer vision @xcite ). these mechanisms draw insight from the intuition that humans build the representation of a whole scene by dynamically focusing on relevant parts at different times @xcite . the general form of attention has been named differently according to authors (alignment model @xcite and attention mechanism @xcite ). in essence, the attention function maps a query q and keys k to scalar scores @xcite . these scores are fed to a softmax function, in turn producing a set of attention weights that are then applied to values v. different kinds of attention are thus possible according to how many keys are attended to (global vs. local attention, according to @xcite ) and where the query is generated (cross vs. self-attention as in the works of @xcite and @xcite ). in this paper, we focus on attention regardless of these technical differences. there are mainly two ways of computing the attention weights α: @xcite introduced additive attention α = softmax(w 3 t tanh(w 1 k + w 2 q)), where w 3 , w 1 , w 2 model parameters to be learned, and @xcite introduced scaled dot-product attention α = softmax kq √ m , where m represents the dimension of k. these two forms are theoretically similar @xcite and generally give the same results @xcite , the dot-product form being faster on certain tasks from a practical point of view. since the introduction of attention mechanisms in the literature, many have seen the opportunity to use the weights for explaining neural networks (e.g., @xcite ; @xcite ; @xcite ; @xcite ; @xcite ). indeed, the attention weights link the input to the remaining of the network with the aim of performing a certain task, and are trained to do so through back-propagation. this link between the input and the remaining of the network is used to work on explainability, which in machine learning and nlp is defined as the capacity to explain a non-interpretable @xcite , i.e., black-box, model @xcite . the two major ways to explain black-box models are global explanations, providing clues about the behavior of the model as a whole, and local explanations, explaining particular decisions. using attention to explain neural networks mainly pertains to the latter, even if some authors study attention for global explanation (e.g., @xcite ). explanations can also be faithful (how close the explanation is to the inner workings of the model) @xcite , or plausible (does the user consider the explanation of the model plausible?) @xcite . it should be noted that explanation presupposes some degree of transparency to the user, whether it is faithful or plausible. indeed, disregarding this aspect would entail that the most faithful explanation is the black-box model itself. recently, a debate fundamentally questioned whether attention can be used as explanation @xcite ). an immediate response by @xcite challenged some of the arguments of @xcite . to this day, the debate about "is attention explanation?" continues and is the source of a rich and diverse literature. researchers from different areas have mostly contributed to this debate without referring to works outside, and sometimes even inside, their area. these insights include theoretical analyses of attention, the necessity to bring users in the loop, questioning the evaluation methodology for model explanation, and more. this paper brings together the papers from these different areas in order to provide an outline of the quickly growing and vast literature on the subject. moreover, we discuss the lessons learned and highlight the main issues and perspectives. to accurately reflect the debate, we only focus on papers that are posterior to the works of @xcite and @xcite , and that explicitly rely on these two papers to contribute to the debate. this paper proposes the first introduction to the debate about "is attention explanation?". the main contributions of this work are as follows: in order to present the different insights on the debate, we briefly summarize the two seminal papers (section 2), describing the arguments of the two original papers that represent the source of the ongoing debate. we also present survey papers that mention the debate within a broader context (section 3). we then investigate the different research perspectives we extracted from the literature (sections 4 to 9). finally, we analyze the insights offered by those works and offer foundations to build upon for future research related to attention as explanation (section 10). the clash between the initial use of attention as explanation and the 2019 studies debating over the validity of considering attention as an expla-nation started a vast literature on the subject. the following section presents survey papers that are mentioning the debate within a broader perspective. ## survey papers mentioning the debate usually, when exploring a question, survey papers are a good starting point, as they have the advantage of covering a broader scope. however, there is no in-depth introduction to the debate, as survey papers only briefly mention the debate and sometimes do not really add something significant for the discussion (e.g., chaudhari et al. (2019) and lindsay (2020) ). please note that we only discuss surveys that add significant elements to the discussion. @xcite propose a survey on attention. they recall the results of @xcite on the fact that attention may not be explanation, but also refer to the fact that only faithful explanations (and not plausible ones; see section 7) are considered. the "explanation" perspective of the survey is focused on the work of @xcite , which discusses how well attention captures the importance of abstract features in multilayer neural networks when dealing with images. @xcite argue that an answer to the question "is attention explanation?" with image data may not generalize to text, and should be verified, as human understanding mechanisms strongly differ between images and texts. contrary to the short introductions to the debate in these survey papers, we aim at providing a clear and rather exhaustive view of the different ways the debate is tackled in the literature. the different insights on the debate, which are unfortunately not regrouped and discussed in these surveys (because the debate is not their main focus), are numerous: some papers add arguments about the fact that attention is not explanation (section 4), provide analyses to explain why attention is not explanation (section 5), analyze the debate on different nlp tasks (section 6), discuss the methodological issues at the heart of the debate (section 7), evaluate the explanatory power of attention with humans (section 8), or propose solutions to make attention become explanation (based on technical developments or on user-in-the-loop strategies) (section 9). ## additional arguments about attention is not explanation some works may be considered as the direct continuation of the arguments of @xcite by adding experiments that corroborate their findings, e.g., by showing that the comparison of attention with other explainable methods different from the gradient-based one leads to similar conclusions. @xcite show that removing features considered as important by attention less often leads to a decision flip than removing features considered important by gradient-based methods. this means that the features deemed important by attention for a decision are not so important for the model. this, therefore, adds to the first argument of @xcite against the relevance of attention as an indicator of feature importance. @xcite demonstrate that applying lime @xcite on an attention-based neural network can provide good explanations that the attention itself cannot provide. they conclude on this subject that their experimental results are aligned with the ones of @xcite . ## analyses of why attention is not explanation in addition to the arguments in the literature on the fact that attention is not explanation, another part of the literature focuses on understanding the reasons why it is not explanation. @xcite show that attention can be put on uninteresting tokens because of an effect they call "combinatorial shortcuts". the key idea is that attention is calculated on the basis of a biased input: "the attention mechanism will try to select biased features to adapt the biased estimations to minimize the overall loss functions" @xcite . for instance, if one adds random tokens (such as a, b, and c) to all documents in a corpus, one might find that some of these tokens are considered as important for the positive (or negative) class because their representation ends up being similar to the representation of "good" (or "bad"), even if their information content for the task is negligible, as they are present in all documents. @xcite theoretically show that attention weights in transformers can be decomposed into two parts, from which the "effective attention" part corresponds to the attention that really affects the output. effective attention focuses on the effective input needed by the model for the task and is not biased by the representation of the input. @xcite extend the work of @xcite , but focus on describing the effective attention part in more detail instead of using it to improve the model. likewise, sun and marasović (2021) also extend the work of @xcite and delve deeper into the explanation of effective attention and its use for explaining the model. @xcite study attention through two specific scores: attention and polarization. the attention score corresponds to the absolute value associated with each input token before the transformation into an attention weight. the polarization score is a global score (not instance-specific) for each input token, indicating its importance for predicting the positive or negative class. the authors show through these two scores why attentionbased models are stable in their prediction, even when attention weights differ. they also show that the match between attention and polarizing scores strongly depends on the hyperparameter values. by analyzing the effect of regularization on attention, @xcite show that one of the reasons why attention cannot be used as a faithful explanation is due to the fact that all input tokens roughly have the same influence on the prediction. the authors show that regularizing attention-based models so that embedded tokens e t better corre-spond to their hidden representation rnn(e t ) produces explanations that are more faithful to the model. however, @xcite show that regularizing generally decreases the correlation between attention and explanation techniques, if the regularization is directed towards sparse attention weights. the authors conclude that sparsity, which is often viewed as increasing interpretability of models in the literature, in this case reduces the faithfulness of explanations. another way to analyze the problem is to study the change in the representation of the meaning of a sentence when (i) an attention layer is added, and when (ii) the type of rnn encoding the input is changed @xcite . the authors show that, in addition to an increase in accuracy, the use of attention also makes the model more stable in terms of representation of sentence meanings. ## is attention explanation on different in this section, we introduce arguments from the literature that claim that, despite some proofs that attention is not always explanation, attention can be explanation on certain nlp tasks. in general, attention mechanisms seem to provide faithful explanations in syntax-related tasks such as part-ofspeech tagging and syntactic annotation. @xcite thus investigate the attention heads in bert in the context of syntactic dependency tagging and co-reference resolution. they find that attention heads at different layers attend to different kinds of information (e.g., direct objects of verbs, determiners of nouns or referential antecedents), with earlier layers having a broader attention span. furthermore, attention heads in the same layer tend to show similar distributions, which is a counter to the argument of @xcite on the fact that encouraging attention heads to learn different distributions within layers can improve performance. overall, knowledge of syntax seems to be encoded by a variety of attention heads in different layers, and thus attention can be used as a global explanation for the tasks under investigation. similarly, @xcite investigate attention in gpt-2, in particular for part-of-speech and syntactic tagging. they find that each partof-speech is attended to by a specific subset of attention heads, and that attention heads in adjacent layers attend to similar part-of-speech tags. in general, attention shows which tokens were attended to for the task at hand and can thus be used as a global explanation. @xcite and @xcite are some of the few works analyzing attention as explanation in a multi-head setting. additional work is needed to establish the similarities and differences between single and multiple heads in the context of the debate. in a different vein, @xcite investigate the role of attention across a variety of nlp tasks. they show that, when the input consists of a single sequence (e.g., in sentiment classification), the attention mechanism is comparable to a gating unit and, as such, the learned weights cannot be interpreted as attention. therefore, in this context, attention does not provide an explanation of the model's reasoning. the reduction of attention to gating units however does not hold true for selfattention networks nor for tasks depending on an additional text sequence, as for example in neural machine translation or natural language inference (pair-wise tasks and text generation tasks). in such cases, altering learned attention weights significantly degrades performance and attention appears to be an explanation of the model and to correlate with feature importance measures. ## evaluation methodology for explanation this section focuses on critics of the methodology when evaluating explanations via attention. the critics mainly focus on two points in the evaluation setup of @xcite . first, @xcite claim that there should be a consistency between attention weights and other explanation methods -which @xcite agree with -and find none. second, they state that the fact that attention could offer different explanations (which they show by shuffling the attention weights) is an issue, which is a strong point of disagreement with @xcite . they conclude that checking for consistency between explanation methods should not be a criterion for evaluation, which goes against the agreement between the two seminal papers. the second point on shuffling the attention weights is a subject of more discussion. @xcite propose a general discussion about logic traps in evaluating interpretation. their take on this point of the debate is that a model with its manipulated attention weights in the work of jain and wallace ( 2019 ) "cannot even be regarded as a trained model, which makes their manipulation meaningless" @xcite , which adds to the point made by @xcite . @xcite argue that it is too early for the debate to take place because there are no good definition and evaluation of explanations. the authors propose a definition driven pipeline (ddp) to evaluate explanations based on the definition of faithfulness. they show that following this ddp can produce an evaluation of explanations that is less biased and can even drive the development of new faithful explanations. calling for more clearly differentiating between faithfulness and plausibility when evaluating explanation, jacovi and goldberg (2020) define five guidelines for evaluating faithfulness, building upon the common pitfalls and sub-optimal practices they observed in the literature. they propose an organization of the literature into three types: model assumption, prediction assumption, and linearity assumption. they state that the distinction between @xcite and @xcite is the underlying assumptions they use for evaluating attention heat-maps as explanations. the former attempts to provide different explanations of similar decisions per instance (therefore linked to prediction assumption). the latter critiques the former and is more anchored in the model assumption type of work. ## evaluating explanations with humans the notion of plausibility of attention-based explanations implies asking humans to evaluate whether attention provides a plausible explanation for the model's decisions. a first issue is whether human judges can agree on what plausible explanations of a decision (e.g., a prediction) are. in an experiment involving predictions for sentiment analysis and reading comprehension, @xcite ask humans to decide whether the top 3 highest weighted words in 200 samples are relevant for the model's prediction. they reported a very high agreement among judges (i.e., cohen's κ over 0.8), which leads to think that words receiving the highest attention can form a plausible explanation. a second interesting issue is the type of human annotations that should be captured in order to assess model's plausibility. the most common approach is to ask humans to assess attention heatmaps produced by a model. in @xcite , users assess the relevance of the top 3 highest weighted words, whereas mohankumar et al. (2020) ask evaluators to decide which of two attention heatmaps better explains the model's prediction as regards to three dimensions: overall prediction, completeness (which heatmap highlights all the words required for the prediction) and correctness (highlights only the important words and not unnecessary words). another way to assess the difference between human and machine attention, in @xcite , consists in asking humans to highlight important words for a classification task. the authors report an agreement percentage around 70% for this task and show that attention weights on top of bi-rnns align pretty well with human attention. this finding is especially true for words for which annotators agree on the importance. a third line of research @xcite uses eye tracking measures to investigate whether machine attention match human attention. the authors hypothesize that machine attention distributions should correlate with human attention strategies for a given task (e.g., question answering). they found that human and machine attention distributions are more similar on easier tasks, which may mean that, for difficult tasks, humans required more varied strategies. for lstms and cnns, diverging more from human attention leads to a drop in performance, which is not the case for xlnets. however, the fact that humans could reliably assess model's plausibility does not ensure that the model is faithful @xcite . in fact, @xcite cast serious doubts on using attention maps as a way for users to audit explanations in the context of fairness. more precisely, the authors train various architectures of neural network models on datasets that are all gender-biased and whose predictions heavily rely on "impermissible" tokens (e.g., pronouns). an adapted loss function is used to penalize the attention values of these impermissible tokens. the authors conclude that, although the problematic tokens are still used by the models, they do not appear in the attention map, which wrongly leads users to believe that the models are unbiased. in other words, the authors proved that a plausible explanation does not always imply that the explanation is faithful. ## solutions to make attention explanation this section proposes an overview of the different solutions that have been developed to tackle the various challenges raised by the debate. we identify two types of solutions: the first type, presented in section 9.1, concerns purely technical solutions that are often based on the theoretical and empirical analyses presented in section 5. the second type of solutions, presented in section 9.2, leverages user-in-the-loop strategies to align machine attention with human attention. ## conclusion we have shown that the debate about the question "is attention explanation?" already produced a vast and diverse literature. throughout our analysis, we highlighted various insights that can help advance the debate: theoretically refining concepts around the notion of explanation (in particular plausibility and faithfulness), developing a common ground in the evaluation setup (e.g., similar input embeddings and architectures), extending the studies and uses of effective attention, and improving the integration of users for a supervised attention. we intend that our work provides a solid ground for further research, calling for more integration to answer the question "is attention explanation?". in particular, combining the findings from the different areas (e.g., to produce a supervised effective attention) seems to be among the most promising avenues.