A Diagnostic Study of Explainability Techniques for Text Classification (2020)

Abstract: Recent developments in machine learning have introduced models that approach human performance at the cost of increased architectural complexity. Efforts to make the rationales behind the models’ predictions transparent have inspired an abundance of new explainability techniques. Provided with an already trained model, they compute saliency scores for the words of an input instance. However, there exists no definitive guide on (i) how to choose such a technique given a particular application task and model architecture, and (ii) the benefits and drawbacks of using each such technique. In this paper, we develop a comprehensive list of diagnostic properties for evaluating existing explainability techniques. We then employ the proposed list to compare a set of diverse explainability techniques on downstream text classification tasks and neural network architectures. We also compare the saliency scores assigned by the explainability techniques with human annotations of salient input regions to find relations between a model’s performance and the agreement of its rationales with human ones. Overall, we find that the gradient-based explanations perform best across tasks and model architectures, and we present further insights into the properties of the reviewed explainability techniques.

## introduction understanding the rationales behind models' decisions is becoming a topic of pivotal importance, as both the architectural complexity of machine learning models and the number of their application domains increases. having greater insight into the models' reasons for making a particular prediction has already proven to be essential for discovering potential flaws or biases in medical diagnosis @xcite and judicial sentencing @xcite . in addition, european law has mandated "the right . . . to obtain an explanation of the decision reached" @xcite . explainability methods attempt to reveal the reasons behind a model's prediction for a single data point, as shown in figure 1 . they can be produced post-hoc, i.e., with already trained models. such post-hoc explanation techniques can be applicable to one specific model @xcite or to a broader range thereof @xcite . they can further be categorised as: employing model gradients @xcite , being perturbation based @xcite or providing explanations through model simplifications @xcite . there also exist explainability methods that generate textual explanations @xcite and are trained posthoc or jointly with the model at hand. while there is a growing amount of explainability methods, we find that they can produce vary-ing, sometimes contradicting explanations, as illustrated in figure 1 . hence, it is important to assess existing techniques and to provide a generally applicable and automated methodology for choosing one that is suitable for a particular model architecture and application task @xcite . @xcite compiles a list of property definitions for explainability techniques, but it remains a challenge to evaluate them in practice. several other studies have independently proposed different setups for probing varied aspects of explainability techniques @xcite . however, existing studies evaluating explainability methods are discordant and do not compare to properties from previous studies. in our work, we consider properties from related work and extend them to be applicable to a broader range of downstream tasks. furthermore, to create a thorough setup for evaluating explainability methods, one should include at least: (i) different groups of explainability methods (explanation by simplification, gradient-based, etc.), (ii) different downstream tasks, and (iii) different model architectures. however, existing studies usually consider at most two of these aspects, thus providing insights tied to a specific setup. we propose a number of diagnostic properties for explainability methods and evaluate them in a comparative study. we consider explainability methods from different groups, all widely applicable to most ml models and application tasks. we conduct an evaluation on three text classification tasks, which contain human annotations of salient tokens. such annotations are available for natural language processing (nlp) tasks, as they are relatively easy to obtain. this is in contrast to ml sub-fields such as image analysis, for which we only found one relevant dataset -536 manually annotated object bounding boxes for visual question answering @xcite . we further compare explainability methods across three of the most widely used model architectures -cnn, lstm, and transformer. the transformer model achieves state-of-the-art performance on many text classification tasks but has a complex architecture, hence methods to explain its predictions are strongly desirable. the proposed properties can also be directly applied to machine learning (ml) subfields other than nlp. the code for the paper is publicly available. 1 1 https://github.com/copenlu/xai-benchmark in summary, the contributions of this work are: ## related work explainability methods can be divided into explanations by simplification, e.g., @xcite ; gradient-based explanations @xcite ; perturbation-based explanations @xcite . some studies propose the generation of text serving as an explanation, e.g., @xcite @xcite . for extensive overviews of existing explainability approaches, see @xcite . explainability methods provide explanations of different qualities, so assessing them systematically is pivotal. a common attempt to reveal shortcomings in explainability techniques is to reveal a model's reasoning process with counter-examples @xcite @xcite , finding different explanations for the same output. however, single counter-examples do not provide a measure to evaluate explainability techniques @xcite . another group of studies performs human evaluation of the outputs of explainability methods @xcite . such studies exhibit low interannotator agreement and reflect mostly what appears to be reasonable and appealing to the annotators, not the actual properties of the method. the most related studies to our work design measures and properties of explainability techniques. @xcite propose an extensive list of properties. the consistency property captures the difference between explanations of different models that produce the same prediction; and the stability property measures the difference between the explanations of similar instances given a single model. we note that similar predictions can still stem from different reasoning paths. instead, we propose to explore instance activations, which reveal more of the model's reasoning process than just the final prediction. the authors propose other properties as well, which we find challenging to apply in practice. we construct a comprehensive list of diagnostic properties tied with measures that assess the degree of each characteristic. another common approach to evaluate explainability methods is to measure the sufficiency of the most salient tokens for predicting the target label @xcite . we also include a sufficiency estimate, but instead of fixing a threshold for the tokens to be removed, we measure the decrease of a model's performance, varying the proportion of excluded tokens. other perturbation-based evaluation studies and measures exist @xcite ), but we consider the above, as it is the most widely applied. another direction of explainability evaluation is to compare the agreement of salient words annotated by humans to the saliency scores assigned by explanation techniques @xcite . we also consider the latter and further study the agreement across model architectures, downstream tasks, and explainability methods. while we consider human annotations at the word level @xcite , there are also datasets @xcite with annotations at the sentence-level, which would require other model architectures, so we leave this for future work. existing studies for evaluating explainability heavily differ in their scope. some concentrate on a single model architecture -bert-lstm (deyoung et al., 2020), rnn @xcite , cnn @xcite , whereas a few consider more than one model @xcite . some studies concentrate on one particular dataset @xcite , while only a few generalize their findings over downstream tasks @xcite . finally, existing studies focus on one @xcite or a single group of explainability methods @xcite . our study is the first to propose a unified comparison of different groups of explainability techniques across three text classification tasks and three model architectures. ## evaluating attribution maps we now define a set of diagnostic properties of explainability techniques, and propose how to quantify them. similar notions can be found in related work (robnik-šikonja and bohanec, 2018; deyoung et al., 2020), and we extend them to be generally applicable to downstream tasks. we first introduce the prerequisite notation. let @xmath0, y i , w i )|@xmath1, n ]} be the test dataset, where each instance consists of a list of tokens x @xmath2,j |@xmath3, |x i |]}, a gold label y i , and a gold saliency score for each of the tokens in x i : let ω be an explanation technique that, given a model m , a class c, and a single instance x i , computes saliency scores for each token in the input: finally, let @xmath4, . . . m k be models with the same architecture, each trained from a randomly chosen seed, and let @xmath5, . . . m k be models of the same architecture, but with randomly initialized weights. agreement with human rationales (ha). this diagnostic property measures the degree of overlap between saliency scores provided by human annotators, specific to the particular task, and the word saliency scores computed by an explainability technique on each instance. the property is a simple way of approximating the quality of the produced feature attributions. while it does not necessarily mean that the saliency scores explain the predictions of a model, we assume that explanations with high agreement scores would be more comprehensible for the end-user as they would adhere more to human reasoning. with this diagnostic property, we can also compare how the type and the performance of a model and/or dataset affect the agreement with human rationales when observing one type of explainability technique. during evaluation, we provide an estimate of the average agreement of the explainability technique across the dataset. to this end, we start at the instance level and compute the average precision (ap) of produced saliency scores ω m confidence indication (ci). a token from a sin-gle instance can receive several saliency scores, indicating its contribution to the prediction of each of the classes. thus, when a model recognizes a highly indicative pattern of the predicted class k, the tokens involved in the pattern would have highly positive saliency scores for this class and highly negative saliency scores for the remaining classes. on the other hand, when the model is not highly confident, we can assume that it is unable to recognize a strong indication of any class, and the tokens accordingly do not have high saliency scores for any class. thus, the computed explanation of an instance i should indicate the confidence p i,k of the model in its prediction. we propose to measure the predictive power of the produced explanations for the confidence of the model. we start by computing the saliency distance (sd) between the saliency scores for the predicted class k to the saliency scores of the other classes k/k (eq. 2). given the distance between the saliency scores, we predict the confidence of the class with logistic regression (lr) and finally compute the mean absolute error -mae (eq. 3), of the predicted confidence to the actual one. for tasks with two classes, d is the subtraction of the saliency value for class k and the other class. for more than two classes, d is the concatenation of the max, min, and average across the differences of the saliency value for class k and the other classes. low mae indicates that model's confidence can be easily identified by looking at the produced explanations. faithfulness (f). since explanation techniques are employed to explain model predictions for a single instance, an essential property is that they are faithful to the model's inner workings and not based on arbitrary choices. a well-established way of measuring this property is by replacing a number of the most-salient words with a mask token @xcite and observing the drop in the model's performance. to avoid choosing an unjustified percentage of words to be perturbed, we produce several dataset perturbations by masking 0, 10, 20, . . . , 100% of the tokens in order of decreasing saliency, thus arriving at x ω 0 , x ω 10 , . . . , x ω 100 . finally, to produce a single number to mea-sure faithfulness, we compute the area under the threshold-performance curve (auc-tp): using auc-tp, we perform an ablation analysis which is a good approximation of whether the most salient words are also the most important ones for a model's prediction. however, some prior studies @xcite find that models remain confident about their prediction even after stripping most input tokens, leaving a few that might appear nonsensical to humans. the diagnostic properties that follow aim to facilitate a more in-depth analysis of the alignment between the inner workings of a model and produced saliency maps. ## results and discussion we report the measures of each diagnostic property as well as flops as a measure of the computing time used by the particular method. for all diagnostic properties, we also include the randomly assigned saliency as a baseline. ## conclusion we proposed a comprehensive list of diagnostic properties for the evaluation of explainability techniques from different perspectives. we further used them to compare and contrast different groups of explainability techniques on three downstream tasks and three diverse architectures. we found that gradient-based explanations are the best for all of the three models and all of the three downstream text classification tasks that we consider in this work. other explainability techniques, such as shapsampl, lime and occlusion take more time to compute, and are in addition considerably less faithful to the models and less consistent with the rationales of the models and similarities in the datasets.