Towards Benchmarking the Utility of Explanations for Model Debugging (2021)

Abstract: Post-hoc explanation methods are an important class of approaches that help understand the rationale underlying a trained model’s decision. But how useful are they for an end-user towards accomplishing a given task? In this vision paper, we argue the need for a benchmark to facilitate evaluations of the utility of post-hoc explanation methods. As a first step to this end, we enumerate desirable properties that such a benchmark should possess for the task of debugging text classifiers. Additionally, we highlight that such a benchmark facilitates not only assessing the effectiveness of explanations but also their efficiency.

## introduction a large variety of post-hoc explanation methods have been proposed to provide insights into the reasons behind predictions of complex machine learning models @xcite . recent work on explainable machine learning in deployment @xcite highlights that explanations are mostly utilized by engineers and scientists to debug models. the use of explanations for model debugging is motivated by their ability to help detect right for the wrong reasons bugs in models. these bugs are difficult to identify from observing predictions and raw data alone and are also not captured by common performance metrics computed on i.i.d. datasets. deep neural networks are particularly vulnerable to learning decision rules that are right for the wrong reasons. they tend to solve datasets in unintended ways by performing shortcut learning @xcite , picking up spurious correlations, which can result in "clever hans behavior" @xcite . considering this important role of explanations during the model validation or selection phase, we call for more utility-focused evaluations of explanation methods for model debugging. we identify two key limitations in current approaches for measuring the utility of explanations for debugging: 1) a ground-truth problem, and 2) an efficiency problem. first, in all common evaluation setups, the presence of bugs serves as a ground truth and although crucial to the evaluation's outcome, intentionally adding bugs to create models that exhibit right for the wrong reasons behavior has not been thoroughly studied. we envision a benchmark collection of verified buggy models to encourage comparable utility-centric evaluations of different explanation methods. bugs can be injected into models by introducing artificial decision rules, so-called decoys, into existing datasets. to establish a rigorous design of decoy datasets, we enumerate desirable properties of decoys for text classification tasks. while a decoy has to be adoptable enough to be verifiably picked up during model training, the resulting decoy dataset should also be natural. second, the utility of explanations is not only determined by their effectiveness. for local explanation methods, i.e., methods that generate explanations for individual instances, the selection of instances examined by humans is crucial to the utility of explanation methods, and thus successful debugging. this efficiency problem of how fast users can detect a bug has been mostly ignored in previous evaluations. by presenting only instances containing a bug they implicitly assume the selection process to be optimal; an assumption that does not transfer to real-world scenarios and potentially leads to unrealistic expectations regarding the utility of explanations. ## evaluating the utility of explanations for debugging the utility of explanations is measured by how useful the explanation is to an end-user towards accomplishing a given task. in this work, we focus on the model developer (as the stakeholder). we outline four different task setups used in previous work. ## ground truth for debugging with explanations in the evaluation approaches presented earlier, we identify crucial components paid little heed in previous work. all the evaluation setups require a model containing one or multiple bugs. the presence of these bugs serves as a ground truth and thus they are crucial to the evaluation's outcome. the bugs introduced into models in the evaluation regimes are "well understood" and added purposely. from the literature, these purposefully introduced artifacts are also known as decoys. although crucial to the evaluation's outcome, these decoys have not been thoroughly studied. as a first step towards a more rigorous design of decoy datasets, we define properties and desiderata for text classification tasks. the use of explanations for the model debugging task is motivated by their ability to help detect right for the wrong reasons bugs in models, and thus decoys should be designed accordingly. ## properties of decoys for text classification in this section, we describe a number of properties and desiderata to consider when designing decoys for text classification tasks. @xcite analyze the nature of spurious statistical unigram and bigram cues contained in the warrants of the argument and reasoning comprehension task (arct) @xcite using three key properties, which we modify for describing token-based decoys in text classification datasets: let x be a dataset of labeled instances (x i , y i ) and x d ⊆ x be the subset of instances containing a decoy d. the applicability a of a decoy d describes the number of instances affected by the decoy, that is, a @xmath0. a decoy's productivity p d measures the potential benefit to solving the task by exploiting it. we define it as the largest proportion of the decoy co-occurring with a certain class label for instances in x d : we further formulate properties that decoys should satisfy for injecting right for the wrong reason bugs: adoptable. discriminative machine learning models typically adopt the decision-rules offering the biggest reward w.r.t minimizing some objective function. if there exists a simpler, more productive decision-rule than the one introduced by the decoy, a model might not adopt the latter and the decoyrule is not learned. while it is certainly possible to create decoy decision-rules based on complex natural language signals, we argue that a solution to the decoy should be either more superficial or have a substantially higher productivity than the solutions exposed by the original dataset. although the potential solutions to a dataset are typically not apparent to humans (otherwise one should probably refrain from using complex machine learning models), researchers and practitioners often have some intuition about the complexity of intended solutions to the task at hand. the adoptability also depends on the decoy being representative. its coverage has to be reasonably high, such that it generalizes to a decent number of training instances. additionally, whether a decoy is adoptable depends on the inductive biases of the model, e.g., a decoy based on word positions is not adoptable by a bag-of-words model. natural. explanations are supposed to help detect right for the wrong reason bugs, which are difficult to identify from observing predictions and raw data alone. it should be possible for a decoy to occur naturally, such that insights from evaluations on decoy datasets can potentially transfer to real-world scenarios. a natural decoy also ensures that humans are not able to easily spot the decoy by observing raw data examples, which would defeat the purpose of using explanations in the first place. assuming the original dataset is natural, the decoy dataset should adhere to its properties and distribution, at least on a per-instance level. for example, for text tasks, the instances affected by a decoy should not violate grammar, syntax, or other linguistic properties, if these are also not violated in the original dataset. the first example in fig. 1 shows an explanation generated for a model trained on a decoy dataset corresponding to the first decoy variant of sst used by @xcite . in this decoy dataset, [cls] it ' s a charming and often affecting journey . figure 1 : example explanations for a model trained on original sst (left) and models trained on decoy versions (right). for all sentences, groundtruth class and predicted class is 'positive'. the input tokens are highlighted based on their contributions towards the prediction, from negative (red) to positive (green) contribution. we finetune bert base @xcite with the default hyperparameter settings recommended in the original paper. the explainer is integrated gradients 1 @xcite . a more natural decoy is shown in rows 2 -5 in fig. 1 , where we create a decoy dataset by removing all instances which contain the word 'movie' and are labeled 'negative', retaining the original dataset's naturalness on a local level. considering all test set instances containing the word 'movie', the performance of a model trained on this decoy dataset drops to random chance (47.5%), indicating that the model was indeed misled by the decoy rule even though its applicability is below 3.3%. ## efficient debugging with explanations another crucial component in the evaluation setups described in section 2 is the choice of instances shown to the human subjects. such a selection is especially important when dealing with large datasets where the majority of instances have correct predictions with explanations aligning with human understanding. showing all instances to humans in order to isolate a few errors is inefficient and often infeasible as the inspection of many individual explanations is expensive in time and resources, especially when requiring domain experts. thus, the examination is typically conducted under a tight budget on the number of instances. when sampling instances of this decoy dataset uniformly, there is a chance of less than 3.3% of being presented with an instance containing the bug. as a result, an evaluation that assumes the selection process to be optimal might not reflect the actual utility of explanations for debugging in practical applications at all. summarizing explanations, for example by spectral relevance clustering @xcite , looks to be a promising way to boost the utility of explanations for tasks like debugging. ## outlook although the current evaluation setups provide a solid foundation, measuring the actual utility of explanations for debugging remains difficult and current evaluations might not transfer to real-world scenarios. we envision a benchmark collection of carefully designed decoy datasets and buggy models to alleviate key limitations and accelerate the future development of new, utility-driven explana-tion methods, as well as methods improving the efficiency of current explanation techniques.