Attention vs non-attention for a Shapley-based explanation method (2021)

Abstract: The field of explainable AI has recently seen an explosion in the number of explanation methods for highly non-linear deep neural networks. The extent to which such methods – that are often proposed and tested in the domain of computer vision – are appropriate to address the explainability challenges in NLP is yet relatively unexplored. In this work, we consider Contextual Decomposition (CD) – a Shapley-based input feature attribution method that has been shown to work well for recurrent NLP models – and we test the extent to which it is useful for models that contain attention operations. To this end, we extend CD to cover the operations necessary for attention-based models. We then compare how long distance subject-verb relationships are processed by models with and without attention, considering a number of different syntactic structures in two different languages: English and Dutch. Our experiments confirm that CD can successfully be applied for attention-based models as well, providing an alternative Shapley-based attribution method for modern neural networks. In particular, using CD, we show that the English and Dutch models demonstrate similar processing behaviour, but that under the hood there are consistent differences between our attention and non-attention models.

## introduction machine learning models using deep neural architectures have seen tremendous performance improvements over the last few years. the advent of models such as lstms @xcite and, more recently, attention-based models such as transformers @xcite have allowed some language technologies to reach near human levels of performance. however, this performance has come at the cost of the interpretability of these models: high levels of nonlinearity make it a near impossible task for a human to comprehend how these models operate. understanding how non-interpretable black box models make their predictions has become an active area of research in recent years @xcite @xcite @xcite . one popular interpretability approach makes use of feature attribution methods, that explain a model prediction in terms of the contributions of the input features. for instance, a feature attribution method for a sentiment analysis task can tell the modeller how much each of the input words contributed to the decision of a particular sentence. multiple methods of assigning contributions to the input feature approaches exist. some are based on local model approximations @xcite , others on gradient-based information @xcite and yet others consider perturbation-based methods (lundberg and lee, 2017) that leverage concepts from game theory such as shapley values @xcite . out of these approaches the shapley-based attribution methods are computationally the most expensive, but they are better able at explaining more complex model dynamics involving feature interactions. this makes these methods well-suited for explaining the behaviour of current nlp models on a more linguistic level. in this work, we therefore focus our efforts on that last category of attribution methods, focusing in particular on a method known as contextual decomposition (cd, @xcite , which provides a polynomial approach towards approximating shapley values. this method has been shown to work well on recurrent models without attention @xcite ), but has not yet been used to provide insights into the linguistic capacities of attentionbased models. here, to investigate the extent to which this method is also applicable for attention based models, we extend the method to include the operations required to deal with attention-based models and we compare two different recurrent models: a multi-layered lstm model (similar to @xcite , and a single headed attention rnn (sha-rnn, @xcite . we focus on the task of language modelling and aim to discover simultaneously whether attribution methods like cd are applicable when attention is used, as well as how the attention mechanism influence the resulting feature attributions, focusing in particular on whether these attributions are in line with human intuitions. following, i.a. jumelet et al. (2019), lakretz et al. (2019) and giulianelli et al. (2018) , we focus on how the models process long-distance subject verb relationships across a number of different syntactic constructions. to broaden our scope, we include two different languages: english and dutch. through our experiments we find that, while both english and dutch language models produce similar results, our attention and non-attention models behave differently. these differences manifest in incorrect attributions for the subjects in sentences with a plural subject-verb pair, where we find that a higher attribution is given to a plural subject when a singular verb is used compared to a singular subject. our main contributions to the field thus lie in two dimensions: on the one hand, we compare attention and non-attention models with regards to their explainability. on the other hand, we perform our analysis in two languages, namely dutch and english, to see if patterns hold in different languages. ## background in this section we first discuss the model architectures that we consider. following this, we explain the attribution method that we use to explain the different models. finally, we consider the task which we use to extract explanations. ## method in this section, we first look at extending contextual decomposition for the sha-rnn. following this, we outline the models which we will use for our experiments. finally, we explain how we extended the number agreement task and how we applied contextual decomposition to the na task, forming the subject attribution task. ## results and analysis in our work, we have considered several experiments. firstly, we evaluate the ability of our models to handle the data itself by comparing the model perplexities. following this, we look at the number agreement and subject attribution tasks to evaluate the differences between our models. ## conclusion in this paper, we compared both attention (sha-rnn) and non-attention (lstm) language models across two languages, namely dutch and english. to test these models, we extended the number agreement task from @xcite to the dutch language, which allows us to compare these models across both languages. in addition to this, we extended a feature attribution method called contextual decomposition @xcite to the sha-rnn model. we applied contextual decomposition to the number agreement task to obtain interpretable explanations and compared the different models from a feature attribution standpoint. we found that both the dutch and english models behaved similarly in terms of accuracy. while general performance differed between the two languages, we did find that similar behavioural patterns emerged from the models. this partially held for the explanations obtained through contextual decomposition, where we did uncover differences. these differences were centred around the sha-rnn, which we found behaved as if it applied default reasoning similar to the work of @xcite . comparing our attention and non-attention models, we found immediate differences, both when comparing the performance on the number agreement task as when looking into the attributions. both models performed differently on the same tasks and feature attributions varied between them. we found that our lstm performed better on the attribution task. our current results suggest that attention and non-attention models behave differently according to contextual decomposition. more specifically, we find that the attention models have more difficulty producing correct attributions for plural sentences. a logical next step would then be to compare our current results by those obtained through different attribution methods such as shap @xcite and integrated gradients @xcite . should we find that contextual decomposition holds up well to these other sees see bias the boy under the tables -0.51 -0.42 0.49 0.39 0.36 0.29 0.31 0.28 0.43 0.36 0.59 0.49 nounpp sp attributions sees see bias the boys under the table -0.21 -0.18 0.34 0.28 0.35 0.3 0.5 0.44 0.36 0.32 0.41 0.35 nounpp ps attributions (a) example sha-rnn attributions verbs verbp bias det subjs prep det nounp -0.44 -0.38 0.51 0.43 0.39 0.33 0.42 0.36 0.5 0.42 0.51 0.43 nounpp sp attributions verbs verbp bias det subjp prep det nouns -0.3 -0.26 0.44 0.38 0.37 0.33 0.37 0.32 0.47 0.41 0.49 0.42 nounpp ps attributions (b) aggregated sha-rnn attributions verbs verbp bias det subjs prep det nounp -2.75 -2.67 0.27 -0.03 0.02 -0.06 -0.29 -0.2 -0.12 -0.09 0.04 0.15 nounpp sp attributions verbs verbp bias det subjp prep det nouns -2.82 -2.65 0.18 0.19 -0.14 0.09 -0.08 -0.02 -0.04 0.0 0.37 0.38 nounpp ps attributions (c) aggregated lstm attributions 3a shows the attributions of two individial sentences, while figs. 3b and 3c show aggregated attributions over all sentences of that condition. note that in fig. 3b the attribution for the subject under the singular verb is both higher in the sp condition as well as in ps condition, while in fig. 3c the attribution is higher for the subject matching the verb form. an alternative line of research that we would like to explore is the attention mechanism itself. even though it has been shown that attention does not provide guarantees for explainability @xcite , it would still be worthwhile to investigate the attention patterns that are employed by the sha-rnn.