The Explanation Game: Towards Prediction Explainability through Sparse Communication (2020)

Abstract: Explainability is a topic of growing importance in NLP. In this work, we provide a unified perspective of explainability as a communication problem between an explainer and a layperson about a classifier’s decision. We use this framework to compare several explainers, including gradient methods, erasure, and attention mechanisms, in terms of their communication success. In addition, we reinterpret these methods in the light of classical feature selection, and use this as inspiration for new embedded explainers, through the use of selective, sparse attention. Experiments in text classification and natural language inference, using different configurations of explainers and laypeople (including both machines and humans), reveal an advantage of attention-based explainers over gradient and erasure methods, and show that selective attention is a simpler alternative to stochastic rationalizers. Human experiments show strong results on text classification with post-hoc explainers trained to optimize communication success.

## introduction the widespread use of machine learning to assist humans in decision making brings the need for explaining models' predictions @xcite @xcite . this poses a challenge in nlp, where current state-ofthe-art neural systems are generally opaque @xcite @xcite . despite the large body of recent work (reviewed in §7), a unified perspective modeling the human-machine interaction-a communication process in its essence-is still missing. many methods have been proposed to generate explanations. some neural network architectures are equipped with built-in components-attention mechanisms-which weigh the relevance of input features for triggering a decision @xcite layperson ŷ m ỹ classifier figure 1: our framework to model explainability as communication. predictions ŷ are made by a classifier c; an explainer e (either embedded in c or operating post-hoc) accesses these predictions and communicates an explanation (a message m) to the layperson l. success of the communication is dictated by the ability of l and c to match their predictions: ỹ ? = ŷ. both the explainer and layperson can be humans or machines. @xcite . top-k attention weights provide plausible, but not always faithful, explanations @xcite @xcite . rationalizers with hard attention are arguably more faithful, but require stochastic networks, which are harder to train @xcite . other approaches include gradient methods @xcite , querying the classifier with leave-one-out strategies @xcite , or training local sparse classifiers @xcite . how should these different approaches be compared? several diagnostic tests have been proposed: @xcite assessed the explanatory power of attention weights by measuring their correlation with input gradients; @xcite and @xcite developed more informative tests, including a combination of comprehensiveness and sufficiency metrics and the correlation with human rationales; @xcite proposed a set of evaluation recommendations and a graded notion of faithfulness. most proposed frameworks rely on correlations and counterfactual simulation, sidestepping the main practical goal of prediction explainability-the ability to communicate an explanation to a human user. in this work, we fill the gap above by proposing a unified framework that regards explainability as a communication problem. our framework is inspired by human-grounded evaluation through forward simulation/prediction, as proposed by doshi-velez (2017, §3.2), where humans are presented with an explanation and an input, and must correctly simulate the model's output (regardless of the true output). we model this process as shown in figure 1 , by considering the interaction between a classifier (the model whose predictions we want to explain), an explainer (which provides the explanations), and a layperson (which must recover the classifier's prediction). we show that different configurations of these components correspond to previously proposed explanation methods, and we experiment with explainers and laypeople being both humans and machines. our framework also inspires two new methods: embedded explainers based on selective attention @xcite , and trainable explainers based on emergent communication @xcite . overall, our contributions are: ## revisiting feature selection a common way of generating explanations is by highlighting rationales @xcite . the principle of parsimony ("occam's razor") advocates simple explanations over complex ones. this principle inspired a large body of work in traditional feature selection for linear models. we draw here a link between that work and modern approaches to explainability. ## embedded sparse attention the case where the explainer e is embedded in the classifier c naturally favors faithfulness, since the mechanism that explains the decision (the why) can also influence it (the how). attention mechanisms @xcite ) allow visualizing relevant input features that contributed to the model's decision. however, the traditional softmax-based attention is dense, i.e., it gives some probability mass to every feature, even if small. the typical approach is to select the top-k words with largest attention weights as the explanation. however, this is not a truly embedded method, but rather a filter, and as pointed out by @xcite and @xcite , it may not lead to faithful explanations. an alternative is to embed in the classifier an attention mechanism that is inherently selective, i.e., which can produce sparse attention distributions natively, where some input features receive exactly zero attention. an extreme example is hard attention, which, as argued by deyoung et al. ( 2020 ), provides more faithful explanations "by construction" as they discretely extract snippets from the input to pass to the classifier. a problem with hard attention is its non-differentiability, which complicates training @xcite . we consider in this paper a different approach: using end-to-end differentiable sparse attention mechanisms, via the sparsemax @xcite and the recently proposed 1.5-entmax transformation @xcite , described in detail in §a. these sparse attention transformations have been applied successfully to machine translation and morphological inflection @xcite . words that receive non-zero attention probability are selected to be part of the explanation. this is an embedded method akin of the use of ℓ 1 -regularization in static feature selection. we experiment with these sparse attention mechanisms in §5. ## explainability as communication we now have the necessary ingredients to describe our unified framework for comparing and designing explanation strategies, illustrated in figure 1 . our fundamental assumption is that explainability is intimately linked to the ability of an explainer to communicate the rationale of a decision in terms that can be understood by a human; we use the success of this communication as a criterion for how plausible the explanation is. ## experiments we experimented with our framework on two nlp tasks: text classification and natural language inference. additional experiments on machine translation are reported in §g, with similar conclusions. we used 4 datasets (sst, imdb, agnews, yelp) for text classification and one dataset (snli) for nli, with statistics and details in classifier c. for text classification, the input @xmath0.g. topics or sentiment labels). the message is a bag of words (bow) extracted from the document. as in @xcite and @xcite , our classifier c is an rnn with attention. for nli, the input x is a pair of sentences (premise and hypothesis) and the labels in y are entailment, contradiction, and neutral. we let messages be again bows, and we constrain them to be selected from the premise (and concatenated with the full hypothesis). we used a similar classifier as above, but with two independent bil-stm layers, one for each sentence. we used the additive attention of @xcite with the last hidden state of the hypothesis as the query and the premise vectors as keys. we also experimented with rnn classifiers that replace softmax attention by 1.5-entmax (c ent ) and sparsemax (c sp ), and with the rationalizer models of @xcite (c bern ) and @xcite (c hk ). details about these classifiers and their hyperparameters are listed in §d. layperson l and explainer e. we used a simple linear bow model as the layperson l. for nli, the layperson sees the full hypothesis, encoding it with a bilstm. the bow from the explainer is passed through a linear projection and summed with the last hidden state of the bilstm. we evaluated the following explainers: we also report a random baseline, which randomly picks k words as the message. we show examples of messages for all explainers in §i. results. the first thing to note is that, as expected, the random baseline is much worse than the other ex-plainers, for all text classification datasets. 4 among the non-trivial explainers, the attention and erasure outperform gradient methods: the erasure and top-k attention explainers have similar csr, with a slight advantage for attention methods. note that the attention explainers have the important advantage of requiring a single call to the classifier, whereas the erasure methods, being wrappers, require k calls. the worse performance of top-k gradient (less severe on agnews) suggests that the words that locally cause bigger output changes are not necessarily the most informative ones. 5 regarding the different attention models (softmax, entmax, and sparsemax), we see that sparse transformations tend to have slightly better acc l , in addition to better acc c (see ## human evaluation to fully assess the quality of the explanations in a more realistic forward simulation setting, we performed human evaluations, where the layperson l is a human instead of a machine. joint training of e and l. so far we compared several explainers, but what happens if we train e and l jointly to optimize csr directly, as described in §4.2? we experiment with the imdb and snli datasets, comparing with using humans for either the layperson, the explainer, or both. human layperson. we randomly selected 200 documents for imdb and snli to be annotated by humans. the extracted explanations (i.e. the selected words) were shuffled and displayed as a cloud of words to two annotators, who were asked to predict the label of each document when seeing only these explanations. for snli, we show the entire hypothesis as raw text and the premise as a cloud of words. the agreement between annotators and other annotation details can be found in §h. human explainer. we also consider explanations generated by humans rather than machines. to this end, we used the e-snli corpus @xcite , which extends the snli with human rationales. since the e-snli corpus does not provide highlights over the premise for neutral pairs, we removed them from the test set. 6 we summarize our results in ## related work there is a large body of work on analysis and interpretation of neural networks. our work focuses on prediction explainability, different from transparency or model interpretability @xcite @xcite . @xcite defines explainability as a plausible reconstruction of the decision-making process, and riedl (2019) argues that it mimics what humans do when rationalizing past actions. this inspired our post-hoc explainers in §4.2 and their use of the faithfulness loss term. recent works questioned the interpretative ability of attention mechanisms @xcite . @xcite distinguished between faithful and plausible explanations and introduced several diagnostic tools. @xcite use human evaluation to show that attention mechanisms produce plausible explanations, consistent with our findings in §6. none of these works, however, considered the sparse selective attention mechanisms proposed in §3. hard stochastic attention has been considered by @xcite ; @xcite ; @xcite ; @xcite , but a comparison with sparse attention and explanation strategies was still missing. besides attention-based methods, many other explainers have been proposed using gradients @xcite @xcite , leave-one-out strategies @xcite , or local perturbations @xcite , but a link with filters and wrappers in the feature selection literature has never been made. we believe the connections revealed in §2 may be useful to develop new explainers in the future. our trained explainers from §4.2 draw inspiration from emergent communication @xcite @xcite . some of our proposed ideas (e.g., using sparsemax for end-to-end differentiability) may also be relevant to that task. our work is also related to sparse auto-encoders, which seek sparse overcomplete vector representations to improve model interpretability @xcite @xcite . in contrast to these works, we consider the non-zero attention probabilities as a form of explanation. some recent work @xcite advocates comprehensive rationales. while comprehensiveness could be useful in our framework to prevent trivial communication protocols between the explainer and layperson, we argue that it is not always a desirable property, since it leads to longer explanations and an increase of human cognitive load. in fact, our analysis of csr as a function of message length (figure 2 ) suggests that shorter explanations might be preferable. this is aligned to the "explanation selection" principle articulated by @xcite : "similar to causal connection, people do not typically provide all causes for an event as an explanation. instead, they select what they believe are the most relevant causes." our sparse, selective attention mechanisms proposed in §3 are inspired by this principle. ## conclusions we proposed a unified framework that regards explainability as a communication problem between an explainer and a layperson about a classifier's decision. we proposed new embedded methods based on selective attention, and post-hoc explainers trained to optimize communication success. in our experiments, we observed that attention mechanisms and erasure tend to outperform gradient methods on communication success rate, using both machines and humans as the layperson, and that selective attention is effective, while simpler to train than stochastic rationalizers.