Goodhart’s Law Applies to NLP ’s Explanation Benchmarks (2024)

Abstract: Despite the rising popularity of saliency-based explanations, the research community remains at an impasse, facing doubts concerning their purpose, efficacy, and tendency to contradict each other. Seeking to unite the community’s efforts around common goals, several recent works have proposed evaluation metrics. In this paper, we critically examine two sets of metrics: the ERASER metrics (comprehensiveness and sufficiency) and the EVAL-X metrics, focusing our inquiry on natural language processing. First, we show that we can inflate a model’s comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution test inputs. Our strategy exploits the tendency for extracted explanations and their complements to be “out-of-support” relative to each other and in-distribution inputs. Next, we demonstrate that the EVAL-X metrics can be inflated arbitrarily by a simple method that encodes the label, even though EVAL-X is precisely motivated to address such exploits. Our results raise doubts about the ability of current metrics to guide explainability research, underscoring the need for a broader reassessment of what precisely these metrics are intended to capture.

## introduction popular methods for "explaining" the outputs of natural language processing (nlp) models operate by highlighting a subset of the input tokens that ought, in some sense, to be salient. the community has initially taken an ad hoc approach to evaluate these methods, looking at select examples to see if the highlighted tokens align with intuition. unfortunately, this line of research has exhibited critical shortcomings @xcite . popular methods tend to disagree substantially in their highlighted token explanations @xcite . other methods highlight tokens that simply encode the predicted label, rather than offering additional information that could reasonably be called an explanation @xcite . this state of affairs has motivated an active area of research focused on developing evaluation metrics to assess the quality of such explanations, focusing on such high-level attributes as faithfulness, plausibility, and conciseness, among others. in particular, faithfulness has emerged as a focus of explainability metrics. according to @xcite , faithfulness "refers to how accurately [an explanation] reflects the true reasoning process of the model." given a prediction model and a saliency method, such metrics are typically concerned with how the prediction model's output changes when it is invoked with only the explanatory tokens or when the model receives the non-explanatory tokens output by the saliency method @xcite @xcite @xcite @xcite . unfortunately, these token subsets typically do not resemble the natural documents the model is trained on. this raises concerns about whether changes in model outputs given these inputs could be due merely to distribution shift @xcite . the design philosophy of evaluating models on out-of-distribution inputs does not originate from these metrics, but instead dates back to the design of many explanation algorithms themselves @xcite . in this paper, we investigate two sets of explanation metrics that rely on evaluating the model on masked inputs: the eraser metrics (i.e. comprehensiveness and sufficiency) and the eval-x metrics. we introduce simple algorithms that wrap existing predictors, and achieve near-optimal scores on these faithfulness metrics without doing anything that a reasonable practitioner might describe as providing better explanations. in the case of the eraser benchmark, we use a simple wrapper model to inflate the faithfulness scores of a original input: "i like this movie. the acting is great." 1. model confidence on the original input: 0.7 for "positive" 2. model confidence on the non-explanatory features for the "positive" predicted label: 3. comprehensiveness score := (1) -( 2 ) a) without score inflation: 0.7 -0.4 = 0.3 b) with score inflation: 0.7 -0.0 = 0.7 (max) a) original prediction model predict normally non-explanation features "i this movie. the" model confidence 0.4 b) our meta-algorithm ## optimizing the eraser benchmark metrics let x denote a sequence of input tokens, @xmath0, . . . , |y|} a categorical target variable, and f a prediction model that maps each input to a predicted probability over the |y| labels. by ŷ, we denote the predicted label, and ê a generated explanation consisting of an ordered subset of the tokens in x. by x \ ê, we denote the non-explanation features that result from deleting the explanation. definition 1 (sufficiency) sufficiency is the difference between the model confidence (on the predicted label) given only the explanation features ## optimizing scores on eval-x metrics the eval-x metrics are focused on the extractthen-classify variety of "explainable" classifiers @xcite . they confront the issue that when an explanation extractor and label predictor are trained jointly, the extractor may end up doing all of the work by simply "encoding" the eventual prediction, rather than providing evidence @xcite . consider for instance, on a binary classification task, an explanation extractor that outputs a period whenever the prediction is positive, and a comma whenever the prediction is negative. the classifier can perfectly recover the predicted label from the single token, encoded explanation. this issue has been highlighted in several past works, including (treviso and martins, 2020) and @xcite . to detect such encoded explanations, @xcite propose evaluating the explanations under the "true" posterior of the predicted label. concretely, what they propose is to train an evaluator model called eval-x. during training, eval-x is given training examples consisting of randomly masked inputs together with the true corresponding target. the proposed evaluation strategy for any explanation method is to assess the accuracy (eacc) and auc (eauroc) of the eval-x model on the masked input consisting only of explanation tokens. @xcite suggest that encoding in extract-then-classify models should be detectable as a sharp decline in eacc and eauroc compared to acc and auroc. we define a sharp decline as a 10% drop. ## conclusion we have demonstrated that simple methods can achieve substantially better and, sometimes, nearoptimal scores on current explanation metrics without producing explanations that anyone would reasonably claim as being more faithful. while these metrics represent honest efforts to codify desider-ata of such explanations, we conclude that they are not suitable to function as benchmarks. in general, few metrics capture all desiderata of interest. accuracy does not capture all desiderata associated with image classification and rouge score is a weak proxy for summarization quality. however, for a quantitative metric to function effectively as a benchmark, concerted efforts to optimize the metric should lead to desired technological improvements. lowering imagenet error, for example, required genuine advancements in computer vision and efforts to increase rouge have revolutionized machine summarization. efforts to optimize a metric, respecting the rules of the game, should not be regarded as mere "gaming"; inspiring such efforts is the very purpose of a benchmark. typically, the development of a metric involves multiple iterations of proposals and critiques before a useful formalism is established. for example, in privacy, many formal notions of privacy were proposed and scrutinized before the community converged on the robust and mathematically rigorous concept of differential privacy while the term explanation may be hopelessly broad, we do not discount the possibility that measures might be proposed that rigorously capture some useful notion of saliency. we hope that these results can inspire improved definitions capable of guiding methodological research.