How (Un)Faithful is Attention? (2022)

Abstract: Although attention weights have been commonly used as a means to provide explanations for deep learning models, the approach has been widely criticized due to its lack of faithfulness. In this work, we present a simple approach to compute the newly proposed metric AtteFa, which can quantitatively represent the degree of faithfulness of the attention weights. Using this metric, we further validate the effect of the frequency of informative input elements and the use of contextual vs. non-contextual encoders on the faithfulness of the attention mechanism. Finally, we apply the approach on several real-life binary classification datasets to measure the faithfulness of attention weights in real-life settings.

## introduction attention mechanism @xcite has become an indispensable part of many state-of-theart nlp models, and its application is becoming more and more prevalent in non-nlp use cases. in simple words and from a functionality perspective, attention can be described as a module which generates outputs from the representations of input elements by performing the following two steps: however, the use of attention weights as explanations has been widely challenged, with regards to the observation that they are not faithful, meaning that different attention weights can result in similar model predictions @xcite @xcite . therefore, the explanations provided by the attention weights are neither unique nor closely related. in this work, we extend the work of @xcite to a one-shot adversarial setup that can be used to compute a quantitative metric for the faithfulness of attention weights. we call the metric attefa which simply stands for attention faithfulness. we consider the adversarial training setup one-shot in the sense that it can provide us with the attefa metric by running the adversarial training only once. to perform a sanity check on attefa, we run experiments in a controlled setting using synthetic datasets and two types of encoders (a noncontextual mlp and a contextual lstm) that could help us validate if the values of this metric reflect what we expect it to. we later compute this metric on some real-life binary text classification datasets to validate how faithful the attention weights are in those settings. ## related work since the rise of deep learning models, researchers have focused on devising techniques that could provide an explanation for the functioning of these so-called "black-box" models. among different classes of explainability techniques, the following can be mentioned: gradient-based methods attribute model decisions to input features using gradient signals @xcite @xcite . perturbation-based methods try to provide an explanation for the model behavior by evaluating its reactions to perturbations in input features @xcite . attention-based methods act as an intuitive way of interpreting the model's decision. they use the probability distribution or weights provided by an attention mechanism as a feature importance measure to find the features that the model is attending to @xcite @xcite . despite the popularity of the attention-based explainability approaches, the reliability of these methods has been called into question, with the special focus on the faithfulness of the explanations provided by the attention mechanism. @xcite perform different experiments to evaluate the meaningfulness of explanations provided by attention weights. their results show that attention weights are not correlated with gradientbased feature importance scores. furthermore, they show that it is often possible to have different attention probability distributions that result in a similar output, arguing that a specific distribution cannot be treated as the definitive cause behind a model decision. @xcite investigate the ability of attention weights to act as importance measures through a different lens. they state that it is not sufficient for the weights to make sense to humans. the weights should also provide a faithful explanation for the model output in order to be considered reliable. through performing multi-weight tests, they show that although there is a certain level of correlation between attention weights and the importance of features in the final prediction of the model, these weights in many cases cannot successfully identify the features that heavily impact a model's decision. @xcite propose additional tests for evaluating the ability of the attention mechanism to provide explainability. they challenged the findings reported by @xcite as they treated the attention as a stand-alone component within a network that is independent from the rest of the components. through an end-to-end adversarial setup to train models to similar outputs while coming up with different attention distributions in binary classification tasks, they show that the explanations provided by attention are not as unfaithful as @xcite found them to be. in this paper, we extend the adversarial setup by @xcite so that it can be used in a one-shot pass, i.e. training the adversar-ial models only once. this approach results in a metric, which we call attefa, that can provide us with a quantitative insight on how faithful the explanations by the attention component are, given a specific model and a specific dataset. to the best of our knowledge, this is the first work that provides such a quantitative measure to evaluate the faithfulness of attention. ## experimental setup the lstm models are comprised of the following components: the mlp models include embedding, attention, and fully-connected modules similar to the lstm models, but utilize a feed-forward projection layer with 128 nodes followed by a tanh activation, instead of the bi-lstm layer. the attention has a two layer fully-connected network that first projects the input to half its size in its first layer, applies a tanh activation, and then maps it to a single logit in the second layer. a softmax function is then used to convert the logit to a probability distribution, which is used to compute a weighted average over the inputs and form the output of the attention. similar to @xcite and wiegreffe and pinter (2019), for the diabetes and anemia datasets, 300d word2vec embeddings @xcite are pre-trained on the combined text from the two datasets. the training is done using cbow with a window size of 10. for the rest of the datasets, 300d publicly-pretrained fasttext embeddings @xcite are used. adam @xcite is used as the optimizer during training, and the learning rate and weight decay rates are set to 1e-3 and 1e-5, respectively. weight decay is applied to every component in the network except the attention module. ## results and discussion first, we have included the f1 scores achieved by the base models in ## limitations there are certain limitations with the current work, in terms of both the methodology used to compute attefa, and the different factors affecting the attention faithfulness. in this section, we explore the ones that we believe are the most important: the current methodology to compute attefa is scoped solely on binary text classification. in order to have attefa as a widely accepted metric in the nlp community, the methodology needs to be extended to other nlp tasks, such as multi-class classification, text retrieval, question answering, machine translation, etc. in the current work, we have studied the effect of the frequency of informative tokens on the faithfulness of attention through running experiments on the mock-* datasets, which are synthetic datasets for sentiment classification. the current selection of sentiment words and their positioning within the input texts were done in a random fashion. a more thorough experiment would explore the effect of the distribution of informative tokens across the input texts (centered towards the start/end/middle vs. scattered evenly), along with a more careful selection of the words to be used as the informative tokens (e.g. differentiating between words with strong vs. weak sentiments). in terms of investigating the effect of encoder contextualization on the faithfulness of attention, we have explored using token-level mlp as a non-contextual encoder and lstm as a contextual one. this can be extended to exploring other encoder architectures, such as cnns @xcite , grus @xcite , and transformers @xcite . another aspect in the current study which has room for exploration is the evaluation of the effect of softmax temperature on the faithfulness of attention. we believe that higher faithfulness may be achieved by using lower temperatures in the case of datasets with infrequent informative tokens, and higher temperature in the case of datasets with frequent informative tokens within their input. last, but not least, the experiments in this work are only focused on a specific type of single-head attention. we believe that the current approach does not transfer properly to multi-headed attentions, as we may still consider a multi-headed attention faithful if the only way for the adversarial model to come up with the same predictions as the base model is to change the order of the attention heads and not the attention weights computed by them. due to the frequent use of multi-headed attentions in state-of-the-art nlp models, the extension of attefa to multi-headed attentions would play a big role in its widespread adoption by the nlp community. ## conclusion in this paper, we presented an adversarial training approach for binary text classification tasks, which can provide us with the metric attefa that quantitatively measures the degree of faithfulness in the attention weights. we, then, measured the effect of contextualization, as well as the effect of the frequency of informative tokens on the attention faithfulness. finally, we computed and evaluated attefa for models trained on several real-life binary text classification datasets. we hope that the presented approach can act as a motivation for researchers to further explore automatic approaches to quantitatively measure the degree of model explainability or its different aspects (e.g. faithfulness, plausibility, sufficiency, etc.). as future directions, we plan to address the limitations specified in section 7 to come up with a more reliable and more widely applicable metric to measure the faithfulness of attention. we also plan to measure attention faithfulness in other settings, e.g. the use of different types of attention such as multi-headed and scaled dot-product @xcite , the use of attention components in different layers of a model, etc.