Staying True to Your Word: (How) Can Attention Become Explanation? (2020)

Abstract: The attention mechanism has quickly become ubiquitous in NLP. In addition to improving performance of models, attention has been widely used as a glimpse into the inner workings of NLP models. The latter aspect has in the recent years become a common topic of discussion, most notably in recent work of Jain and Wallace; Wiegreffe and Pinter. With the shortcomings of using attention weights as a tool of transparency revealed, the attention mechanism has been stuck in a limbo without concrete proof when and whether it can be used as an explanation. In this paper, we provide an explanation as to why attention has seen rightful critique when used with recurrent networks in sequence classification tasks. We propose a remedy to these issues in the form of a word level objective and our findings give credibility for attention to provide faithful interpretations of recurrent models.

## introduction not long since its introduction, the attention mechanism @xcite has become a staple of many nlp models. apart from enhancing prediction performance of models and starting the trend of fully attentional networks @xcite , attention weights have been widely used as a method for interpreting decisions of neural models. recently, the validity of interpreting the decision making process of a model through its attention weights came under question. @xcite introduced a set of experiments on english language sequence classification tasks which demonstrated that attention weights do not correlate with feature importance measures, and that attention weights generated by a trained model can be substituted and modified without detriment to model performance. while it is natural to assume that multiple plausible explanations for a model's decision can coexist, the authors show the existence of attention distributions that assign most of their mass to words seemingly irrelevant to the task, while still not affecting neither the decision nor the confidence of the model. in the follow-up work, @xcite find that, while such adversarial attention distributions do exist, they are seldom converged to in the training process, even when one introduces a training signal with the sole purpose of guiding the model to such distributions. in this paper, we aim to tackle the difficult question of the relationship between attention and explanation from a different angle -is there any modification we can make to the existing models so that attention could be reliably used as a tool of model transparency? for the sake of consistency, we follow previous work @xcite and limit our scope to single-sequence binary classification tasks, where we consider models from the rnn + self-attention family. concretely, we analyse single-layer bidirectional lstm-s @xcite equipped with the additive @xcite and dot-product @xcite selfattention mechanisms. inspired by the recent results @xcite , which show that optimizing the masked language modelling (mlm) @xcite objective results in high mutual information between the input and output layers of models, we ask ourselves whether such a trait is beneficial for interpretability. the task of sequence classification in no way incentivizes a model to retain information from the input, and the model is likely to filter out information irrelevant to the task. 1 we believe this lack of enforced information retention causes a discrepancy between the input and hidden vectors, which results in reduced model interpretability. to enforce information retention, we propose a number of techniques to keep the hidden representations closer to their input representations, improving the faithfulness of interpreting models through inspecting their attention weights. the contributions of this paper are as follows: we (1) investigate whether the lack of a word-level objective causes attention not to be a faithful interpretation, (2) propose various regularization methods in order to improve interpretability through inspecting attention weights, and (3) quantitatively and qualitatively evaluate whether and how these methods help model interpretability. the rest of the paper is organized as follows. firstly ( §2), we position ourselves within current work and discuss the use of attention as interpretation in nlp,. we then ( §3) present our experimental setup, introduce various regularization methods, and briefly describe the experiments we use to evaluate our regularized models. in §4, we offer a quantitative evaluation of the effect of regularizes on the trained models across a number of datasets. we then ( §5) qualitatively and quantitatively inspect the effect of regularization on a trained model, identifying what we believe to be the cause of negative results reported in previous work. finally ( §6), we summarize our findings and propose possible lines of future work. ## attention and interpretability in nlp preliminaries: let the input sequence of word embeddings be denoted as {w t } t @xmath0, where t is the length of the sequence. the sequence of hidden states produced by the encoder is then {h t } t @xmath1, where each h @xmath2, h (t-1) ). the rnn used is a bidirectional lstm. when discussing a hidden state h t , we refer only to x t as its input for convenience. the attention mechanism produces a probability distribution over the hidden states, the elements of which we denote {α t } t @xmath3, and refer to as attention weights. ## experimental setup the base model used in @xcite ) is a single-layer bidirectional lstm augmented with either a dotproduct or an additive attention mechanism, the output of which is then fed into a linear classifier (decoder). we use the same base model as a baseline throughout our experiments. ## understanding the effect of model regularization to visually demonstrate the undesired effect of attention mechanisms when trained in the base setting, as well as to illustrate the effect of regularizations we applied, we first analyse how we obtain the classifier prediction. the output of the classifier is an affine transformation of the attention output: we can reformulate this as a convex attentionweighted sum of logits (p t ) obtained by running each individual hidden state through the decoder. once we scale the logits for individual timesteps, we obtain the prediction probability as if the whole attention mass was on that hidden representation. for attention weights to be a faithful measure of interpretability, this probability should be high only on tokens which are decision-relevant. in fig. 2 , we plot these token-level probabilities for a single example to demonstrate that in the base model, this is not the case. we can see that for the base model, the probabilities for most tokens have nearly the same probability as the final prediction, while the regularization keeps the representations for neutral words grounded closer to the decision boundary. as a direct result of this, the model predictions are much more fragile to change of attention weights, as only a small number of hidden states are far enough from the decision boundary to produce an equally confident classification. ## conclusion we have identified the lack of a word-level objective as the likely cause of attention weights not being a faithful tool of interpretability in the case of sequence classification with attention mechanism augmented recurrent networks. we experimentally establish that we can add regularization methods to sequence classification which strengthen the relationship between the input and hidden states while not being a detriment to classification performance. if one wishes to interpret classifier decisions through inspecting attention weights, we strongly suggest inclusion of a technique such as weight tying or adding masked language modelling as an auxiliary. adding such methods causes the model to become more susceptible to attacks modifying the attention weights of a trained model, and increases faithfulness of explanations produced by attention weights. while we believe our work is a step forward towards using attention weights as a faithful explanation, by no means do we claim that the modification is sufficient. as was our primary concern, the risk with using attention weights as a tool of interpretability is that a single bad explanation could have consequences in decision-making scenarios, and while our methods improve the faithfulness of such interpretability, it is by no means foolproof. we have only scratched the surface of faithful interpretability, and most of the datasets in our and previous work do not have human annotated rationales. in order to fully understand the cases in which attention provides a reliable explanation, we believe that datasets with annotated rationales or decision-relevant tokens should be used. this analysis should also be extended to more complex models which better capture the nuances of language. we believe that the experiments we presented demonstrate the shortcomings of interpreting model decisions through inspecting attention weights, however we acknowledge that this branch of research sorely lacks evaluation methods that include humans in the loop.