BERT meets Shapley: Extending SHAP Explanations to Transformer-based Classifiers (2021)

Abstract: Transformer-based neural networks offer very good classification performance across a wide range of domains, but do not provide explanations of their predictions. While several explanation methods, including SHAP, address the problem of interpreting deep learning models, they are not adapted to operate on state-of-the-art transformer-based neural networks such as BERT. Another shortcoming of these methods is that their visualization of explanations in the form of lists of most relevant words does not take into account the sequential and structurally dependent nature of text. This paper proposes the TransSHAP method that adapts SHAP to transformer models including BERT-based text classifiers. It advances SHAP visualizations by showing explanations in a sequential manner, assessed by human evaluators as competitive to state-of-the-art solutions.

## introduction recent wide spread use of deep neural networks (dnns) has increased the need for their transparent classification, given that dnns are black box models that do not offer introspection into their decision processes or provide explanations of their predictions and biases. several methods that address the interpretability of machine learning models have been proposed. model-agnostic explanation approaches are based on perturbations of inputs. the resulting changes in the outputs of the given model are the source of their explanations. the explanations of individual instances are commonly visualized in the form of histograms of the most impactful inputs. however, this is insufficient for text-based classifiers, where the inputs are sequential and structurally dependent. we address the problem of incompatibility of modern explanation techniques, e.g., shap @xcite , and state-of-the-art pretrained transformer networks such as bert @xcite . our contribution is twofold. first, we propose an adaptation of the shap method to bert for text classification, called transshap @xcite . second, we present an improved approach to visualization of explanations that better reflects the sequential nature of input texts, referred to as the transshap visualizer, which is implemented in the transshap library. the paper is structured as follows. we first present the background and motivation in section 2. section 3 introduces transshap, an adapted method for explaining transformer language model such as bert, which includes the transshap visualizer for improved visualization of the generated explanations. section 4 presents the results of an evaluation survey, followed by the discussion of results and the future work in section 5. ## background and motivation we first present the transformer-based language models, followed by an outline of perturbationbased explanation methods, in particular the shap method. we finish with the overview of visualizations for prediction explanations. bert @xcite is a large pretrained language model based on the transformer neural network architecture @xcite . nowadays, bert models exist in many mono-and multilingual variants. fine-tuning bert-like models to a specific task produces state-of-the-art results in many natural language processing tasks, such as text classification, question answering, pos-tagging, dependency parsing, inference, etc. there are two types of explanation approaches, general and model specific. the general explanation approaches are applicable to any prediction model, since they perturb the inputs of a model and observe changes in the model's output. the second type of explanation approaches are specific to certain types of models, such as support vector machines or neural networks, and exploit the internal information available during training of these methods. we focus on general explanation methods and address their specific adaptations for use in text classification, more specifically, in text classification with transformer models such as bert. the most widely used perturbation-based explanation methods are ime @xcite , @xcite , and shap (lundberg and lee, 2017). their key idea is that the contribution of a particular input value (or set of values) can be captured by 'hiding' the input and observing how the output of the model changes. in this work, we focus on the stateof-the-art explanation method shap (shapley additive explanations) that is based on the shapley value approximation principle. lundberg and lee (2017) noted that several existing methods, including ime and lime, can be regarded as special cases of this method. we propose an adaptation of shap for bertlike classifiers, but the same principles are trivially transferred to lime and ime. to understand the behavior of a prediction model applied to a single instance, one should observe perturbations of all subsets of input features and their values, which results in exponential time complexity. @xcite showed that the contribution of each variable corresponds to the shapley value from the coalition game, where players correspond to input features, and the coalition game corresponds to the prediction of an individual instance. shapley values can be approximated in time linear to the number of features. the visualization approaches implemented in the explanation methods lime and shap are primarily designed for explanations of tabular data and images. although the visualization with lime includes adjustments for text data, the resulting explanations are presented in the form of histograms that are sometimes hard to understand, as figure 1 shows. the visualization with shap for the same sentence is illustrated in figure 2 . here, the fea-tures with the strongest impact on the prediction correspond to longer arrows that point in the direction of the predicted class. for textual data this representation is non-intuitive. various approaches have been proposed to interpret neural text classifiers. some of them focus on adapting existing shap based explanation methods by improving different aspects, e.g., the word masking (chen and ji, 2020), or reducing feature dimension @xcite , while others explore the complex interactions between words (contextual decomposition) that are crucial when dealing with textual data but are ignored by other post-hoc explanation methods @xcite . ## evaluation we evaluated the novel visualization method using an online survey. the targeted respondents were researchers and phd students not involved in the study that mostly had some previous experience with classifiers and/or their explanation methods. in the survey, the respondents were presented with three visualization methods on the same example: two visualizations were generated by existing libraries, lime and shap, and the third one used our novel transshap library. respondents were asked to evaluate the quality of each visualization, suggest possible improvements, and rank the three methods. 3 the results of 38 completed surveys are as follows. the most informative features of the visualization layout recognized by the users were the impact each word had on a prediction and the importance of the word contributions shown in a sequential view. the positioning of the visualization elements for each of the three methods was rated on the scale of 1 to 5. our method achieved the highest average score of 3.66 (63.1% of the respondents rated it with a score of 4 or 5), second best was the lime method with an average score of 3.13 (39.1% rated it with 4 or 5), and the shap method was rated as the worst with an average of 2.42 (81.5% rated it with 1 or 2). regarding the question whether they would use each visualization method, lime scored highest (44.7% voted "yes"), transshap closely followed (42.1% voted "yes"), while shap was not praised (34.2% voted "yes"). the overall ranking also corresponds to these results. lime got the most votes (54.3%), transshap was voted second best (40.0% of votes), and shap was the least desirable (5.7% of votes). in addition, we asked the participants to choose the preferred usage of the method out of the given options. the transshap and shap methods were considered most useful for the purpose of debugging and bias detection, while the lime method was also recognized as suitable for explaining a model to other researchers (usage in scientific articles). ## conclusion and further work we presented the transshap library, an extension of the shap explanation approach for transformer 3 the survey questions are available here: https:// forms.gle/icpyvhh78oe2tcjt7 . neural networks. transshap offers a novel testing ground for better understanding of neural text classifiers, and will be freely accessible after acceptance of the paper (for review purposes available here: https://bit.ly/2uvy2dy ). the explanations obtained by transshap were quantitatively compared in a user survey, where we assessed the visualization capabilities, showing that the proposed transshap's visualizations were simple, yet informative when compared to existing instance-based visualizations produced by lime or shap. transshap was scored better than shap, while lime was scored slightly better in terms of overall user preference. however, in specific elements, such as positioning of the visualization elements, the visualization produced by transshap is slightly better. in further work, we plan to address problems of the perturbation-based explanation process when dealing with textual data. currently, transshap only supports random sampling from the word space, which may produce unintelligible and grammatically wrong sentences, and overall completely uninformative texts. we intend to take into account specific properties of text data and apply language models in the sampling step of the method. we plan to restrict the sampling candidates for each word based on their part of speech and general context of the sentence. we believe that better sampling will improve the speed of explanations and decrease the variance of explanations. furthermore, the explanations could be additionally improved by expanding the features of explanations from individual words to larger textual units consisting of words that are grammatically and semantically linked.