“Will You Find These Shortcuts?” A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification (2022)

Abstract: Feature attribution a.k.a. input salience methods which assign an importance score to a feature are abundant but may produce surprisingly different results for the same model on the same input. While differences are expected if disparate definitions of importance are assumed, most methods claim to provide faithful attributions and point at the features most relevant for a model’s prediction. Existing work on faithfulness evaluation is not conclusive and does not provide a clear answer as to how different methods are to be compared.Focusing on text classification and the model debugging scenario, our main contribution is a protocol for faithfulness evaluation that makes use of partially synthetic data to obtain ground truth for feature importance ranking. Following the protocol, we do an in-depth analysis of four standard salience method classes on a range of datasets and lexical shortcuts for BERT and LSTM models. We demonstrate that some of the most popular method configurations provide poor results even for simple shortcuts while a method judged to be too simplistic works remarkably well for BERT.

## introduction a prominent class of explainability techniques assign salience scores to the input features, which reflect the importance of the features to the model's decision. when applied to text classifiers those methods produce highlights over the input (sub)words. interestingly, different methods may produce surprisingly dissimilar highlights. figure 1 shows this using the language interpretability tool @xcite . so a natural question is: which method should one use? while a method whose highlights happen to look plausible may facilitate a task like text annotation @xcite @xcite , many salience methods seem to be figure 1 : salience maps produced by four common methods on a sentiment classification example (sst2) for a bert model. the same token (eastwood) is assigned the highest (grad-l2), the lowest (gxi, lime) and a mid-range (ig) importance score (color intensity indicates salience; blue and purple stand for positive, red stands for negative weights). a developer investigating a hypothesis about specific named entities being associated with the label would probably be unsure as to whether the example provides support for or against the hypothesis. spurious correlations plague nlp models too @xcite @xcite @xcite -notorious examples are the tendencies of nli classifiers to overrely on negation or identity words when predicting contradiction and toxicity, respectively @xcite . importantly, shortcuts can comprise multiple tokens. for example, @xcite and @xcite demonstrated that bert sentiment classifiers trained on imdb @xcite learn to largely ignore the review text when patterns like '3 out of 10' or '7 / 10' are present -that is, when the numeric rating is made explicit in the text. making such lexical shortcuts apparent to the developer is thus a strong use case for faithful input salience methods which would then indeed help them improve both the model and the data. how can we know if a method consistently places the shortcut tokens on top of its salience rankings? evaluating this is challenging, because we usually do not know the shortcut in advance and the model parameter space is large. moreover, we don't have an inherently interpretable view into the predictions of common black-box neural models. glass-box models with explicit mediating factors @xcite are not widely used or are synthetic, and model-native structures such as attention have been shown to have weak predictive power @xcite ). alternatively, one can make strong assumptions about what a ground truth should be like and compare salience rankings with what is expected to be the ground truth. in this vein human reasoning @xcite @xcite , gradient information @xcite , aggregated model internal representations @xcite , changes in predicted probabilities @xcite or surrogate models @xcite all have been taken as a proxy for the ground truth when evaluating salience methods. unfortunately, they also resulted in divergent recommendations so the question of what the ground truth is and which method to use remains open. unlike the cited work we argue for a faithfulness evaluation methodology which makes use of partially synthetic data to obtain the ground truth and which is moreover also contextualized in a debugging scenario @xcite . towards the goal of identifying salience methods which would be most helpful in revealing shortcuts learned by a model we make the following contributions: ## methodology we desire two properties from any faithful salience method which is claimed to be helpful for model debugging: high precision and low rank, which we define as follows: precision@k is a measure over the top-k tokens in a salience ranking where k is the shortcut size. with s, m and x i denoting a salience method, a trained model m and the ith example from the synthetic set d and assuming two functions, top k (•) foot_2 and gt k (•) which output the top-k tokens from a salience ranking and the ground truth, respectively: in our experiments (sec. 2.2), k is fixed for a dataset: @xmath0. however, the metric can be trivially adjusted if k varies between dataset instances. mean rank represents how deep, on average, we need to go in a salience ranking to cover all the ground truth tokens: define shortcut type augment dataset with shortcuts train models on original and augmented data verify shortcuts run salience methods compute rankings step 1 step 2 step 3 step 4 step 5 step 6 figure 2: the proposed protocol to obtain ground truth importance rankings. intuitively, precision tells us how many of the important tokens we will find if we focus on the top of the ranking while rank indicates how much of the ranking is needed to find all the important tokens. ## experimental setup we use three text classification datasets and apply the three shortcuts presented above to each of them. despite all the datasets being binary and of comparable size, there are a few differences which may affect a salience method's performance: in the results section we use the following format to refer to a dataset-shortcut combination: sst2:tic, imdb:op, toxicity:st, etc. foot_5 ## results in this section we highlight our main findings. for increased readability where rank scores support precision, we omit them in the main paper and instead present them in the appendix a.4. all the results reported in the paper are computed from a single model checkpoint and a single run. a method's performance varies across model and shortcut types and other dataset properties. it is apparent that gxi performs quite well for lstm models but does not work at all for bert models (tab. 2). conversely, grad l2 performs very well for bert but not at all so for lstm grad {l|p}×l * is a good choice for bert but not lstm models for finding shortcuts. for bert models, grad l2 achieves high precision and rank scores across the different datasets and shortcut types, yielding 0.99 or higher on seven out of nine datasets (tab. 3 and 7 ). the lowest but still comparatively high precision (0.87) is on imdb:tic where the inputs are particularly long. for lstm models, on six out of nine datasets the precision of the same method is around .5 ( in tab. 3 ). it does not matter whether probabilities or logits are used and whether l1 or l2 norm is applied. we hypothesize that one reason for the difference in performance between bert and lstm is that bert models have residual connections, making the gradient information less noisy. however, the results of grad mean are very poor, ranging between .3 and .4 in precision ( in tab. 3 ). note that grad l2 is sometimes deemed unsuitable because it is unsigned and only returns positive scores @xcite , but our experiments demonstrate that it is the most useful method for finding lexical shortcuts learned by bert. using probabilities instead of logits only changes the results for ig. for other gradientbased methods it does not seem to make a large difference. ( and in tab. 4 and 8). ig performance does not improve much with more steps. increasing the number of interpolation steps from 100 to 1000 does not result in a significant improvement for lstm models. also for bert, the precision numbers improve only for the tic shortcuts and only when probabilities are used (last two rows in tab. 4 and 8). the similarity of the scores between the gxi and ig when using the zero baseline ( in tab. 4 and 8) indicates that there is no difference between taking a single or 100(0) steps from the zero baseline. sst2 p imdb p toxicity p st tic op st tic op st tic op lstm ig l-zero-{100|1000} 7 1. .72 .83 .99 .67 .80 1. .95 .95 ig l-unk-{100|1000} 1. .87 .71 1. .71 .79 1. .71 .78 ig l-pad-{100|1000} 1. .77 .85 .99 .67 .79 1 .68 .67 ig p-zero-{100|1000} .93 .69 .68 .78 .66 .76 1. .93 .87 ig p-unk-{100|1000} 1. .82 .77 1. .70 .63 1. .64 .63 ig p-pad-{100|1000} .95 .74 .78 .83 .67 .75 bert gxi {p|l} .29 .58 .31 .59 .35 .50 .41 .43 .47 ig l-zero-{100|1000} .29 .58 .31 .59 .35 .50 .41 .43 .47 ig l-mask-{100|1000} .71 .58 .71 .99 .62 .61 .69 .50 .47 ig l-pad-{100|1000} .79 .27 .14 .28 .47 .27 .36 .46 .18 ig p-zero-{100|1000} .29 .58 .31 .59 .35 .50 .41 .43 .47 ig p-mask-100 .48 .37 .56 .80 .34 .48 .27 .27 .29 ig p-mask-1000 .48 .48 .56 .80 .47 .48 .28 .29 .29 ig p-pad-{100|1000} .81 .18 .1 .21 .31 .14 .16 .37 .12 sst2 p imdb p toxicity p st tic op st tic op st tic op lstm lime unk-100 .98 .80 .83 .92 .50 .62 .93 .58 .59 lime unk-1000 1. .83 .85 .99 .66 .78 1. .84 .66 lime unk-3000 1. .84 .85 1. .66 .80 1. .85 .66 bert lime unk-100 .89 .80 .71 .91 .62 .44 .67 .54 .51 lime unk-1000 .97 .87 .77 .99 .75 .70 .98 .58 .75 lime unk-3000 .98 .88 .77 .99 .77 .71 1. .59 .78 lime mask-3000 .98 .62 .78 .93 .76 .67 .99 .58 .70 choice of baseline is important for ig when using bert. for the most part using the [mask] baseline (with logits) resulted in an improvement in the scores (bold rows in tab. 4 and 8). still, even with the best performing configuration of ig the results are much worse than grad l-l2 . number of perturbations as well as masking token matter for lime. lime benefits from 1000 over 100 perturbations, especially for longer inputs and/or shortcuts. we found that the increase from 1000 to 3000 perturbations leads to little precision improvements for the input lengths in our datasets. using unk for masking leads to better results than @xcite in almost all configurations ( in tab. 5 and 9). we hypothesize this is due to two reasons: (i) the [mask] token is not used during fine-tuning on the task data. (ii) the unk token, however, is finetuned (due to unknown tokens and as special token in word dropout). erasing tokens leads, on average, to worse precision results than masking, for all number of perturbations. tables 10 and 11 in appendix a.5 present the results for all the models, shortcut types and source datasets in terms of precision and rank and you can observe this phenomenon there. rank and precision give complementary information. for example, the precision of grad l2 and grad mean is close on toxicity:op (.56 and .60) while the rank of the latter is almost twice as big (13 and 21) ( in tab. 3 ). lower rank with comparable precision means that the method consistently puts one of the shortcut tokens on the top but buries the other token deep in the ranking. ## related work research on input salience methods for text classification is prolific and diverse in terms of the definitions used @xcite , applications @xcite , desiderata @xcite , etc. the importance of getting faithful salience explanations has been recognized early on @xcite and there exist formal definitions of explanation fidelity @xcite . however, these have not been connected to model debugging where it is the top of a salience ranking that matters most. in the vision domain, our work is closest to @xcite , who also explore the debugging scenario with salience maps, @xcite , who use synthetic data to obtain the ground truth for pixel importance, and @xcite , who contrast the performance of the same model trained on original and modified data when evaluating feature importance. as pointed out in introduction, in nlp faithfulness evaluation has often been grounded in strong assumptions @xcite @xcite or by analyzing models substantially different from the ones normally used @xcite ). an exception to this trend is the work by @xcite who also modify source data but, unlike us, consider mlp as the only dnn model, do not evaluate any gradient-based methods and analyze single token shortcuts only without strong guarantees of them actually being the most important clues for the model. also @xcite analyze dnn models on intentionally corrupted data: they primarily focus on vision but also run an experiment analyzing how faithfully the attention mechanism points at the words known to correlate with the label. finally, @xcite , following @xcite , iteratively remove tokens to evaluate faithfulness of salience methods for lstm models and conclude, similar to us, that performance is task-dependent. concurrently with our work, idahl et al. (2021) argue for faithfulness evaluation on synthetic data for model debugging but do not report experimental results. similarly to them and also concurrently with our work, @xcite go further and combine data and input attribution methods to discover data artifacts. however, citing prior work, they use grad l-mean and ig l-mean which, as we have shown, are sub-optimal configurations for bert models. this explains the very poor accuracy of 12-13% (in our terms: precision@1) that they observed when discovering single-token shortcuts in sst2. finally, as our experiments demonstrate, the single-token shortcut is insufficient to assess whether a method would be useful for more complex shortcuts. ## conclusions we have argued for evaluating input salience methods with respect to how helpful they would be for discovering shortcuts that are learned by the model. this seems to be a clear use case from the model developer perspective. to achieve this, we proposed a protocol for method evaluation and applied it to three variants of lexical shortcuts (single token, token in context, and ordered pair) which are a proxy for shortcut heuristics that occur in common nlp tasks and which are particularly suitable for being discovered with input salience methods. by comparing the performance across different datasets, shortcut types and models (lstm-based and bert-based), we demonstrated that a strong performance for one setup may not hold for a different model or a more complex shortcut. finally, we pointed out that some method configurations assumed to be reliable in recent work, for example integrated gradients, may give very poor results for nlp models, and that the details of how the methods are used can matter a lot, such as how a gradient vector is reduced into a scalar. our results demonstrate that whenever one uses bert and is interested if a simple token combination could determine the label, one should prefer grad-l2 over more complex methods.