Is Sparse Attention more Interpretable? (2021)

Abstract: Sparse attention has been claimed to increase model interpretability under the assumption that it highlights influential inputs. Yet the attention distribution is typically over representations internal to the model rather than the inputs themselves, suggesting this assumption may not have merit. We build on the recent work exploring the interpretability of attention; we design a set of experiments to help us understand how sparsity affects our ability to use attention as an explainability tool. On three text classification tasks, we verify that only a weak relationship between inputs and co-indexed intermediate representations exists—under sparse attention and otherwise. Further, we do not find any plausible mappings from sparse attention distributions to a sparse set of influential inputs through other avenues. Rather, we observe in this setting that inducing sparsity may make it less plausible that attention can be used as a tool for understanding model behavior.

## introduction interpretability research in natural language processing (nlp) is becoming increasingly important as complex models are applied to more and more downstream decision making tasks. in light of this, many researchers have turned to the attention mechanism, which has not only led to impressive performance improvements in neural models, but has also been claimed to offer insights into how models make decisions. specifically, a number of works imply or directly state that one may inspect the attention distribution to determine the amount of influence each input token has in a model's decision-making process @xcite @xcite . many lines of work have gone on to exploit this assumption when building their own "interpretable" models or analysis tools @xcite @xcite ; one subset has even tried to make models with attention more interpretable by inducing sparsity-a common attribute of interpretable models @xcite -in attention weights, with the motivation that this allows model decisions to be mapped to a limited number of items @xcite @xcite . yet, there lacks concrete reasoning or evidence that sparse attention weights leads to more interpretable models: customarily, attention is not directly over the model's inputs, but rather over some representation internal to the model, e.g. the hidden states of a recurrent network or contextual embeddings of a transformer (see fig. 1 ). importantly, these internal representations do not solely encode information from the input token they are co-indexed with @xcite , but rather from a range of inputs. this presents the question: if internal representations themselves may not be interpretable, can we actually deduce anything from "interpretable" attention weights? we build on the recent line of work challenging the validity of attention-as-explanation methods @xcite @xcite , inter alia) and specifically examine how sparsity affects their observations. to this end, we introduce a novel entropy-based metric to measure the dispersion of inputs' influence, rather than just their magnitudes. through experiments on three text classification tasks, utilizing both lstm and transformer-based models, we observe how sparse attention affects the results of @xcite and @xcite , additionally exploring whether it allows us to identify a core set of inputs that are important to models' decisions. we find we are unable to identify such a set when using sparse attention; rather, it appears that encouraging sparsity may simultaneously encourage a higher degree of contextualization in intermediate representations. we further observe a decrease in the correlation between the attention distribution and input feature importance measures, which exacerbates issues found by prior works. the primary conclusion of our work is that we should not believe sparse attention enhances model interpretability until we have concrete reasons to believe so; in this preliminary analysis, we do not find any such evidence. ## attention-based neural networks we consider inputs @xmath0. we denote the embedding of x, e.g., its one hot encoding or (more commonly) a linear transformation of its one-hot encoding with an embedding matrix e 2 r d⇥|v| , as x (e) 2 r d⇥n . our embedded input x (e) is then fed to an encoder, which produces n intermediate representa- an alignment function a(•, •) maps a query q and a key k to weights a (t) for a decoding time step t; we subsequently drop t for simplicity. in colloquial terms, a chooses which values of k should receive the most attention based on q, which is then represented in the vector a (t) 2 r n . for the nlp tasks we consider, we have @xmath1; . . . ; h n ], the encoder outputs. a query q may be, e.g., a representation of the question in question answering. the weights a are projected to sum to 1, which results in the attention distribution ↵. mathematically, this is done via a projection onto the probability simplex using a projection function , e.g., softmax or sparsemax. we then compute the context vector as @xmath2. this context vector is fed to a decoder, whose structure is again architecture dependent, which generates a (possibly unnormalized) probability distribution over the set of labels y, where y is defined by the task. attention. we experiment with two methods of constructing an attention distribution: (1) additive attention, proposed by @xcite : ## model interpretability model interpretability and explainability have been framed in different ways @xcite as model understanding tasks, where (spurious) features learned by a model are identified, or as decision understanding tasks, where explanations for particular instances are produced. we consider the latter in this paper. such tasks can be framed as generative, where models generate free text explanations @xcite @xcite , or as post-hoc interpretability methods, where salient portions of the input are highlighted @xcite @xcite . as there does not exist a clearly superior choice for framing decision understanding for nlp tasks @xcite @xcite , we follow a substantial body of prior work in considering the post-hoc definition of interpretability based on local methods proposed by @xcite . this definition is naturally operationalized through feature importance metrics and meta models @xcite . further, we acknowledge the specific requirement that an interpretable model obeys some set of structural constraints of the domain in which it is used, such as monotonicity or physical constraints @xcite . for nlp tasks such as sentiment analysis or topic classification, such constraints may logically include the utilization of only a few key words in the input when making a decision, in which case, knowing the magnitude of the influence each input token has on a model's prediction through, e.g., feature importance metrics, may suffice to verify the model obeys such constraints. while this collective definition is limited @xcite @xcite , we posit that if attention cannot provide model interpretability at this level, then it would likewise not be able to under more rigorous constraints. ## conclusion prior work has cited interpretability as a driving factor for promoting sparsity in attention distributions. we explore how induced sparsity affects the ability to use attention as a tool for explaining model decisions. in our experiments on text classification tasks, we see that while sparse attention distributions may allow us to pinpoint influential intermediate representations, we are unable to find any plausible mapping from sparse attention to a small, critical set of influential inputs. rather, we find evidence that inducing sparsity may make it even less plausible to use the attention distribution to interpret model behavior. we conclude that we need further reason to believe sparse attention increases model interpretability as our results do not support such claims.