Learning to Deceive with Attention-Based Explanations (2020)

Abstract: Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing. In addition to yielding gains in predictive accuracy, attention weights are often claimed to confer interpretability, purportedly useful both for providing insights to practitioners and for explaining why a model makes its decisions to stakeholders. We call the latter use of attention mechanisms into question by demonstrating a simple method for training models to produce deceptive attention masks. Our method diminishes the total weight assigned to designated impermissible tokens, even when the models can be shown to nevertheless rely on these features to drive predictions. Across multiple models and tasks, our approach manipulates attention weights while paying surprisingly little cost in accuracy. Through a human study, we show that our manipulated attention-based explanations deceive people into thinking that predictions from a model biased against gender minorities do not rely on the gender. Consequently, our results cast doubt on attention’s reliability as a tool for auditing algorithms in the context of fairness and accountability.

## introduction since their introduction as a method for aligning inputs and outputs in neural machine translation, attention mechanisms @xcite have emerged as effective components in various neural network architectures. attention works by aggregating a set of tokens via a weighted sum, where the attention weights are calculated as a function of both the input encodings and the state of the decoder. because attention mechanisms allocate weight among the encoded tokens, these coefficients are sometimes thought of intuitively as indicating which tokens the model focuses on when making a particular prediction. based on this loose intuition, attention weights are often claimed to explain a model's predictions. for example, a recent survey on attention @xcite remarks: in another work, @xcite study gender bias in machine learning models for occupation classification. as machine learning is increasingly used in hiring processes for tasks including resume filtering, the potential for bias raises the spectre that automating this process could lead to social harms. @xcite use attention over gender-revealing tokens (e.g., 'she', 'he', etc.) to verify the gender bias in occupation classification models-stating that "the attention weights indicate which tokens are most predictive". similar claims about attention's utility for interpreting models' predictions are common in the literature @xcite @xcite @xcite . in this paper, we question whether attention scores necessarily indicate features that influence a model's predictions. through a series of experiments on diverse classification and sequence-tosequence tasks, we show that attention scores are surprisingly easy to manipulate. we design a simple training scheme whereby the resulting models appear to assign little attention to a specified set of impermissible tokens while continuing to rely upon those features for prediction. the ease with which attention can be manipulated without significantly affecting performance suggests that even if a vanilla model's attention weights conferred some insight (still an open and ill-defined question), these insights would rely on knowing the objective on which models were trained. our results present troublesome implications for proposed uses of attention in the context of fairness, accountability, and transparency. for example, malicious practitioners asked to justify how their models work by pointing to attention weights could mislead regulators with this scheme. for instance, looking at manipulated attention-based explanation in lastly, practitioners often overlook the fact that attention is typically not applied over words but over final layer representations, which themselves capture information from neighboring words. we investigate the mechanisms through which the manipulated models attain low attention values. we note that (i) recurrent connections allow information to flow easily to neighboring representations; (ii) for cases where the flow is restricted, models tend to increase the magnitude of representations corresponding to impermissible tokens to offset the low attention scores; and (iii) models additionally rely on several alternative mechanisms that vary across random seeds (see §5.3). ## related work many recent papers examine whether attention is a valid explanation or not. @xcite iden-tify alternate adversarial attention weights after the model is trained that nevertheless produce the same predictions, and hence claim that attention is not explanation. however, these attention weights are chosen from a large (infinite up to numerical precision) set of possible values and thus it is not surprising that multiple weights produce the same prediction. moreover since the model does not actually produce these weights, they would never be relied on as explanations in the first place. similarly, @xcite modify attention values of a trained model post-hoc by hard-setting the highest attention values to zero. they find that the number of attention values that must be zeroed out to alter the model's prediction is often too large, and thus conclude that attention is not a suitable tool to for determining which elements should be attributed as responsible for an output. in contrast to these two papers, we manipulate the attention via the learning procedure, producing models whose actual weights might deceive an auditor. in parallel work to ours, @xcite examine the conditions under which attention can be considered a plausible explanation. they design a similar experiment to ours where they train an adversarial model, whose attention distribution is maximally different from the one produced by the base model. here we look at a related but different question of how attention can be manipulated away from a set of impermissible tokens. using human studies we show that our training scheme leads to attention maps that are more deceptive, since people find them to be more believable explanations of the output (see §5.2). we also extend our analysis to sequenceto-sequence tasks, and a broader set of models, including bert, and identify mechanisms by which the manipulated models rely on the impermissible tokens despite assigning low attention to them. lastly, several papers deliberately train attention weights by introducing an additional source of supervision to improve predictive performance. in some of these papers, the supervision comes from known word alignments for machine translation @xcite , or by aligning human eye-gaze with model's attention for sequence classification @xcite . ## manipulating attention let @xmath0, w 2 , . . . , w n denote an input sequence of n words. we assume that for each task, we are dataset (task) input example impermissible tokens (percentage) commoncrawl biographies (physician vs surgeon) ms. x practices medicine in memphis, tn and is affiliated with . . . ms. x speaks english and spanish. gender indicators (6.5%) wikipedia biographies (gender identification) after that, austen was educated at home until she went to boarding school with cassandra early in 1785 gender indicators (7.6%) sst + wikipedia (sentiment analysis) good fun, good action, good acting, good dialogue, good pace, good cinematography. helen maxine lamond reddy (born 25 october 1941) is an australian singer, actress, and activist. sst sentence (45.5%) reference letters (acceptance prediction) it is with pleasure that i am writing this letter in support of . . . i highly recommend her for a place in your institution. percentile:99.0 rank:extraordinary. percentile, rank (1.6%) further, let α ∈ [0, 1] n denote the attention assigned to each word in s by a model, such that i α @xmath1. for any task-specific loss function l, we define a new objective function @xmath2. for a single attention layer, we define r as: and λ is a penalty coefficient that modulates the amount of attention assigned to impermissible tokens. the argument of the log term (1 -α t m) captures the total attention weight assigned to permissible words. in contrast to our penalty term, @xcite use kl-divergence to maximally separate the attention distribution of the manipulated model (α new ) from the attention distribution of the given model (α old ): @xmath3. when dealing with models that employ multiheaded attention, which use multiple different attention vectors at each layer of the model @xcite we can optimize the mean value of our penalty as assessed over the set of attention heads h as follows: when a model has many attention heads, an auditor might not look at the mean attention assigned to certain words but instead look head by head to see if any among them assigns a large amount of attention to impermissible words. anticipating this, we also explore a variant of our approach for manipulating multi-headed attention where we penalize the maximum amount of attention paid to impermissible words (among all heads) as follows: for cases where the impermissible set of tokens is unknown apriori, one can plausibly use the top few highly attended tokens as a proxy. ## experimental setup we study the manipulability of attention on four binary classification problems, and four sequenceto-sequence tasks. in each dataset, (in some, by design) a subset of input tokens are known a priori to be indispensable for achieving high accuracy. ## results and discussion in this section we examine how lowering attention affects task performance ( § 5.1). we then present experiments with human participants to quantify the deception with manipulated attention ( § 5.2). lastly, we identify alternate workarounds through which models preserve task performance ( § 5.3). ## conclusion amidst practices that perceive attention scores to be an indication of what the model focuses on, we characterize the manipulability of attention mechanism and the (surprisingly small) cost to be paid for it in accuracy. our simple training scheme produces models with significantly reduced attention mass over tokens known a priori to be useful for prediction, while continuing to use them. further analysis reveals how the manipulated models cheat, and raises concerns about the potential use of attention as a tool to audit models.