Towards Transparent and Explainable Attention Models (2020)

Abstract: Recent studies on interpretability of attention distributions have led to notions of faithful and plausible explanations for a model’s predictions. Attention distributions can be considered a faithful explanation if a higher attention weight implies a greater impact on the model’s prediction. They can be considered a plausible explanation if they provide a human-understandable justification for the model’s predictions. In this work, we first explain why current attention mechanisms in LSTM based encoders can neither provide a faithful nor a plausible explanation of the model’s predictions. We observe that in LSTM based encoders the hidden representations at different time-steps are very similar to each other (high conicity) and attention weights in these situations do not carry much meaning because even a random permutation of the attention weights does not affect the model’s predictions. Based on experiments on a wide variety of tasks and datasets, we observe attention distributions often attribute the model’s predictions to unimportant words such as punctuation and fail to offer a plausible explanation for the predictions. To make attention mechanisms more faithful and plausible, we propose a modified LSTM cell with a diversity-driven training objective that ensures that the hidden representations learned at different time steps are diverse. We show that the resulting attention distributions offer more transparency as they (i) provide a more precise importance ranking of the hidden states (ii) are better indicative of words important for the model’s predictions (iii) correlate better with gradient-based attribution methods. Human evaluations indicate that the attention distributions learned by our model offer a plausible explanation of the model’s predictions. Our code has been made publicly available at https://github.com/akashkm99/Interpretable-Attention

## analyzing attention mechanisms here, we first investigate the question -why attention distributions may not provide a faithful explanation for the model's predictions? we later examine whether attention distributions can provide a plausible explanation for the model's predictions, not necessarily faithful. ## orthogonal and diversity lstm based on our previous argument that high conicity of hidden states affect the transparency and explainability of attention models, we propose 2 strategies to obtain reduced similarity in hidden states. ## analysis of the model we now analyse the proposed models by performing experiments using the tasks and datasets described earlier. through these experiments we establish that (i) the proposed models perform comparably to vanilla lstms (sec. 5.2) (ii) the attention distributions in the proposed models provide a faithful explanation for the model's predictions (secs. 5.3 to 5.5) and (iii) the attention distributions are more explainable and align better with a human's interpretation of the model's prediction (secs. 5.6, 5.7). throughout this section we will compare the following three models: 1. vanilla lstm: the model described in section 2.1 which uses the vanilla lstm. 2. diversity lstm: the model described in section 2.1 with the vanilla lstm but trained with the diversity objective described in section 4.2. 3. orthogonal lstm: the model described in section 2.1 except that the vanilla lstm is replaced by the orthogonal lstm described in section 4.1. ## related work our work in many ways can be seen as a continuation to the recent studies @xcite @xcite on the subject of interpretability of attention. several other works @xcite @xcite @xcite focus on improving the interpretability of attention distributions by inducing sparsity. however, the extent to which sparse attention distributions actually offer faithful and plausible explanations haven't been studied in detail. few works @xcite map attention distributions to human annotated rationales. our work on the other hand does not require any additional supervision. work by @xcite focus on developing interpretable lstms specifically for multivariate time series analysis. several other works @xcite @xcite @xcite analyze attention distributions and attention heads learned by transformer language models. the idea of orthogonalizing representations in an lstm have been used by @xcite but they use a different diversity model in the context of improving performance of natural language generation models ## conclusion & future work in this work, we have analyzed why existing attention distributions can neither provide a faithful nor a plausible explanation for the model's predictions. we showed that hidden representations learned by lstm encoders tend to be highly similar across different timesteps, thereby affecting the interpretability of attention weights. we proposed two techniques to effectively overcome this shortcoming and showed that attention distributions in the resulting models provide more faithful and plausible explanations. as future work, we would like to extend our analysis and proposed techniques to more complex models and downstream tasks.