Double Trouble: How to not Explain a Text Classifier’s Decisions Using Counterfactuals Synthesized by Masked Language Models? (2022)

Abstract: A principle behind dozens of attribution methods is to take the prediction difference between before-and-after an input feature (here, a token) is removed as its attribution. A popular Input Marginalization (IM) method (Kim et al., 2020) uses BERT to replace a token, yielding more plausible counterfactuals. While Kim et al., 2020 reported that IM is effective, we find this conclusion not convincing as the Deletion-BERT metric used in their paper is biased towards IM. Importantly, this bias exists in Deletion-based metrics, including Insertion, Sufficiency, and Comprehensiveness. Furthermore, our rigorous evaluation using 6 metrics and 3 datasets finds no evidence that IM is better than a Leave-One-Out (LOO) baseline. We find two reasons why IM is not better than LOO: (1) deleting a single word from the input only marginally reduces a classifier’s accuracy; and (2) a highly predictable word is always given near-zero attribution, regardless of its true importance to the classifier. In contrast, making LIME samples more natural via BERT consistently improves LIME accuracy under several ROAR metrics.

## introduction feature attribution maps (ams), i.e. highlights indicating the importance of each input token w.r.t. a classifier's decision, can help improve human accuracy on downstream tasks including detecting fake movie reviews @xcite or identifying biases in text classifiers @xcite . many leave-one-out (loo) methods compute the attribution of an input token by measuring the prediction changes after substituting that token's embedding with zeros @xcite or [unk] @xcite . that is, deleting or replacing features is the underlying principle of at least 25 attribution methods @xcite . ## methods and related work let f : r n×@xmath0, 1] be a text classifier that maps a sequence x of n token embeddings, each of size d, onto a confidence score of an output label. an attribution function a takes three inputs-a sequence x, the model f , and a set of hyperparameters hand outputs a vector @xmath1, x, h) ∈ [-1, 1] n . here, the explanation a associates each input token x i to a scalar a @xmath2, 1], indicating how much x i contributes for or against the target label. leave-one-out (loo) is a well-known method @xcite @xcite for estimating the attribution a i by computing the prediction-difference after a token x i is left out of the input x, creating a shorter sequence x -i : under @xcite causal framework, the attribution a i in eq. 1 relies on a single, unrealistic counterfactual x -i and thus is a biased estimate of the individual treatment effect (ite): ## bias of deletion metric and its variants in explaining sst-2 classifiers, we successfully reproduce the auc rep results reported in @xcite , i.e. im outperformed loo zero and loo unk , which were implemented by replacing a word with the [pad] and [unk] token of bert, respectively ( to test this hypothesis, we add another baseline of loo empty , which was not included in @xcite , i.e. erasing a token from the input without replacement (eq. 1), mirroring the original deletion metric. to compare with im, all loo methods in this paper are also in the log-odds space. results interestingly, we find that, under deletion, on both sst-2 and e-snli, im underperformed all three loo baselines and that loo empty is the highest-performing method ( re-running the same experiment but sampling replacement words from roberta (instead of bert), we find the same finding that loo empty is the best under deletion while im is the best under deletion bert ( to our knowledge, our work is the first to document this bias of the deletion metric widely used in the literature @xcite @xcite . this bias, in principle, also exists in other deletion variants including insertion @xcite , sufficiency, and comprehensiveness @xcite . ## no evidence that im is better than loo to avoid the critical bias of deletion and deletion bert , we further compare im and loo on four common metrics that are not deletion-based. ## discussion and conclusion we find in sec. 5.3 that im is highly insensitive to classifier's changes because, by design, it always assigns near-zero attribution to highly-predictable words x i regardless of their true importance to a target classifier. a solution may be to leave such x i token out of the marginalization (eq. 3), i.e. only marginalizing over the other tokens suggested by bert. however, these other replacement tokens altogether have a sum likelihood of 0. that is, replacing token x i by zero-probability tokens (i.e. truly implausible) would effectively generate ood text, which, in turn is not desired @xcite . our results in sec. 6.2 suggests that im might be more useful at the phrase level @xcite instead of word level as deleting a set of contiguous words has a larger effect to the classifier predictions. in sum, for the first time, we find that the popular idea of harnessing bert to generate realistic counterfactuals @xcite @xcite does not actually improve upon a simple loo empty in practice as an loo empty counterfactual only has a single word deleted. in contrast, we observe more expected benefits of this technique in improving methods like lime that has counterfactuals that are extremely syntactically erroneous when multiple words are often deleted.