Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Models (2024)

Abstract: Despite the widespread adoption of autoregressive language models, explainability evaluation research has predominantly focused on span infilling and masked language models. Evaluating the faithfulness of an explanation method—how accurately it explains the inner workings and decision-making of the model—is challenging because it is difficult to separate the model from its explanation. Most faithfulness evaluation techniques corrupt or remove input tokens deemed important by a particular attribution (feature importance) method and observe the resulting change in the model’s output. However, for autoregressive language models, this approach creates out-of-distribution inputs due to their next-token prediction training objective. In this study, we propose a technique that leverages counterfactual generation to evaluate the faithfulness of attribution methods for autoregressive language models. Our technique generates fluent, in-distribution counterfactuals, making the evaluation protocol more reliable.

## introduction most modern nlp systems rely on autoregressive, transformer-based language models @xcite @xcite . these models are inherently opaque, creating a strong need to understand their decision-making processes. as a result, explanation methods have become increasingly important in the field. a widely-used approach for model explainability is attribution, also known as feature importance (fi) @xcite . attribution methods aim to identify which input features contribute most to a model's predictions, assigning a scalar value to each feature that reflects its relevance in the decision-making process. in typical nlp tasks, input features are often subwords or their combinations. a key challenge in evaluating the faithfulness of attribution methods is that many existing techniques are designed for denoising or masked language models (mlms) @xcite @xcite @xcite . recent work on autoregressive models has primarily focused on the plausibility of attributions @xcite . while plausible (or persuasive) explanations might be the objective of the explainer, the core objective for the user is to truly understand the model's decision-making process, rather than simply being convinced that the model's decisions are correct @xcite . nearly all previous methods for faithfulness evaluation modify the input in some way, such as masking or removing important tokens based on the attribution results, and then measuring the impact on the model's predictions. these methods tend to work well for mlms, which are specifically trained for tasks like span or mask infilling. however, in the case of autoregressive models like gpt-2, which predict the next token, such modifications produce out-of-distribution (ood) inputs. this raises a crucial question: are these evaluation methods truly assessing the informativeness of the selected tokens, or merely testing the model's robustness to unnatural text and the artifacts introduced by testing modifications @xcite ? moreover, the ood nature of these inputs results in explanations that become socially misaligned @xcite . in other words, the expectations of users-who seek to understand which features are most relevant to the model's decision-no longer align with the actual output of the attribution method. instead, feature importance becomes influenced by the model's priors rather than the learned features that truly drive predictions. in this work, drawing inspiration from coun-terfactual generation-where the input is altered to flip the model's output-we propose a new technique to evaluate the faithfulness of attribution methods in autoregressive language models. specifically, we use counterfactual generators to modify the input by focusing on tokens highlighted by attribution methods, while ensuring that the altered input remains natural, fluent, and within the model's original distribution. this ensures that any observed change in the model's predictions is due to the modification of the important tokens, rather than an effect of ood inputs. we argue that if an attribution method enables a counterfactual generator to modify fewer tokens to change the model's prediction, then it demonstrates a stronger understanding of the model's inner workings, indicating higher faithfulness. to validate our approach, we apply this faithfulness evaluation technique to several attribution methods-including gradient norm, gradient × input, erasure, kernelshap, and integrated gradients-within the context of next-word prediction for two language models: the fine-tuned gemma-2b and the off-the-shelf gemma-2b-instruct @xcite . our contributions are as follows: (i) we introduce a novel faithfulness evaluation protocol that preserves the model's input distribution, designed for attribution methods in autoregressive language models. (ii) we apply this protocol to evaluate and rank widely-used attribution methods, showcasing differences in sensitivity between fine-tuned and off-the-shelf models when handling ood data and proposing a solution. foot_0 2 related work evaluating explanations. most current metrics for evaluating faithfulness involve either removing important tokens or retraining the model using only those identified as important by attribution methods @xcite . for instance, abnar and zuidema (2020) assess explanations by comparing them with gradient and ablation techniques. although @xcite caution that gradients should not be considered ideal or the "ground truth," they still utilize gradients as a proxy for the model's intrinsic semantics. importantly, the trustworthiness of explanations is both taskand model-dependent @xcite , and different attribution methods frequently produce inconsistent results @xcite . as a result, it is not justifiable to treat any single explanation method as a universal standard across all contexts. in their work, deyoung et al. ( 2020 ) introduce two key concepts: comprehensiveness (whether the important tokens identified are the only ones necessary for making a prediction) and sufficiency (whether these important tokens alone are enough to make the prediction). carton et al. (2020) build on this by proposing normalized versions of these concepts, comparing comprehensiveness and sufficiency to the null difference-the performance of an empty input (for sufficiency) or a full input (for comprehensiveness). however, it remains unclear whether these corruption techniques evaluate the informativeness of the corrupted tokens or merely the robustness of the model to unnatural inputs and artifacts introduced during evaluation. further, @xcite and jain et al. (2020) frame attribution methods as either faithful or unfaithful, with no consideration for degrees of faithfulness. they describe attribution methods that are "faithful by construction." in contrast, other researchers propose that faithfulness exists on a spectrum and suggest evaluating the "degree of faithfulness" of explanation methods @xcite . our approach aligns with this view, as we aim to find explanation methods that are sufficiently faithful for autoregressive models. @xcite evaluate the faithfulness of natural language explanations using counterfactuals, applying techniques from @xcite to assess how well explanations align with the model's decision-making. this line of work offers valuable insights into the use of counterfactuals, which we build upon for evaluating attribution methods in language models. another relevant direction is the evaluation of explanations using uncertainty estimation. for example, @xcite develop a bayesian framework that generates feature importance estimates along with their associated uncertainty, expressed through credible intervals, highlighting the importance of uncertainty in faithfulness evaluations the ood problem in explainability. the issue of ood inputs in explainability has been raised by several works. @xcite and @xcite suggest retraining or finetuning the model using partially erased inputs to align training and evaluation distributions. however, this process can be computationally expen-sive and is not always practical. an alternative approach by @xcite aims to ensure that the explanation remains in-distribution to mitigate ood problems. our work addresses this concern by preserving the input distribution during faithfulness evaluation, particularly for autoregressive models. feature importance (attribution). attributions, or feature importance scores, are local explanations that assign a score to each input feature-typically token embeddings in nlp tasks-indicating how crucial that feature is to the model's prediction. attribution methods can be categorized into four types: i) perturbation-based methods, which alter or mask input features to assess their importance by observing changes in the model's output @xcite @xcite . ii) gradient-based methods, which calculate the derivative of the model's output with respect to each input to measure the influence of each feature @xcite @xcite @xcite @xcite . iii) surrogatebased methods, which explain a complex black-box model using a simpler, interpretable model @xcite @xcite . iv) decomposition-based methods, which break down the overall importance score into linear contributions from the input features @xcite @xcite @xcite . ## our method our faithfulness evaluation protocol involves two models: a counterfactual generator model and a predictor model. our goal is to evaluate the faithfulness of attribution methods for the predictor model. due to the large output space of autoregressive language models (lms), which often includes thousands of vocabulary items, examining the entire output space does not provide much insight. therefore, we use the contrastive explanations proposed by @xcite , which measure the attribution of input tokens for a contrastive model decision. contrastive attributions aim to identify the most important tokens that led the model to predict the target y t instead of a foil y f . we then use a separate editor model to modify these important tokens to generate counterfactuals-examples that make the original predictor model more likely to predict the foil. our protocol for evaluating attributions consists of two phases. the first phase involves creating the editor that can generate counterfactuals. in the second phase, we use the editor and predictor together to determine what percentage of tokens the editor needs to change to flip the predictor model's prediction. figure 2 illustrates the second phase. to create the editor, we fine-tune an autoregressive language model specifically for counterfactual generation. during fine-tuning, we add two tokens to the embedding space and the tokenizer: '<@xmath0. inspired by @xcite and @xcite , we create training examples for our counterfactual generator by randomly masking between 5% and 50% of the tokens. we then append each example's label (e.g., positive or negative for the sst-2 dataset), the '<@xmath1, and finally the original unmasked example. the process of creating training examples is shown in figure 3 . in the second phase of evaluating attributions, we first input a sentence into the predictor and apply an attribution method to identify the most important tokens influencing the predictor's decisionmaking process. we begin by replacing 10% of these most important tokens with '<@xmath2. if unsuccessful in flipping the prediction, we incrementally increase the masking by 10% until we either flip the prediction or reach a masking threshold of 50%. this evaluation protocol is depicted in figure 2 . the prompting technique used for counterfactual generation during this phase is shown in figure 1 . the attribution technique that identifies the most critical tokens for creating counterfactuals and enables counterfactuals with the least amount of change to the original text is considered to provide the most faithful representation of the predictor's decision-making process. figure 2: our process of generating counterfactuals for evaluating attribution methods. the predictor (an lm) generates a label for the given text, and an attribution method specifies the most important tokens. we mask the top n% of them and ask an editor (another lm) to change the label of the input text by filling in the masked tokens. if the attribution method is more faithful, then the required n% should be lower. 4 experimental setup 4.1 datasets we use three datasets for evaluating faithfulness: sst-2 @xcite and imdb @xcite , which are both binary classification datasets, and ag-news @xcite , a four-class classification dataset. faithfulness evaluation datasets should not have gold attribution labels because we do not want human intuition to influence the evaluation. instead, we aim to understand how the model makes predictions @xcite . ## conclusion in this work, we designed a faithfulness evaluation protocol based on counterfactual generation. we demonstrated that the efficacy of attribution methods varies between models fine-tuned on our specific dataset and off-the-shelf, instruct-tuned models. we showed that counterfactual generators are effective for evaluating feature attribution because they can produce mostly in-distribution text for the predictor model. this approach allows us to separate the evaluation of the model from the evaluation of the attribution method, as the examples used are mostly in-distribution. we also found high consistency between different counterfactual generators and a lack of consistency with other replacement methods, highlighting the importance of being in-distribution, particularly when evaluating attributions on off-the-shelf models. finally, we used our protocol to compare different attribution methods.