Evaluating Explainable AI : Which Algorithmic Explanations Help Users Predict Model Behavior? (2020)

Abstract: Algorithmic approaches to interpreting machine learning models have proliferated in recent years. We carry out human subject tests that are the first of their kind to isolate the effect of algorithmic explanations on a key aspect of model interpretability, simulatability, while avoiding important confounding experimental factors. A model is simulatable when a person can predict its behavior on new inputs. Through two kinds of simulation tests involving text and tabular data, we evaluate five explanations methods: (1) LIME, (2) Anchor, (3) Decision Boundary, (4) a Prototype model, and (5) a Composite approach that combines explanations from each method. Clear evidence of method effectiveness is found in very few cases: LIME improves simulatability in tabular classification, and our Prototype method is effective in counterfactual simulation tests. We also collect subjective ratings of explanations, but we do not find that ratings are predictive of how helpful explanations are. Our results provide the first reliable and comprehensive estimates of how explanations influence simulatability across a variety of explanation methods and data domains. We show that (1) we need to be careful about the metrics we use to evaluate explanation methods, and (2) there is significant room for improvement in current methods.

## introduction interpretable machine learning is now a widely discussed topic @xcite @xcite . while survey papers have not converged on definitions of "explainable" or "interpretable," there are some common threads in the discourse. commentators observe that interpretability is useful for achieving other model desiderata, which may include building user trust, identifying the influence of certain variables, understanding how a model will behave on given inputs, and ensuring that models are fair and unbiased. in their review, @xcite outline an approach to measuring interpretability. they describe two human-subject tasks that test for a particularly useful property: simulatability. a model is simulatable when a person can predict its behavior on new inputs. this property is especially useful since it indicates that a person understands why a model produces the outputs it does. the first of the two tasks is termed forward simulation: given an input and an "explanation," users must predict what a model would output for the given input. the second is counterfactual simulation: users are given an input, a model's output for that input, and an "explanation" of that output, and then they must predict what the model will output when given a perturbation of the original input. the explanation itself is algorithmically generated by a method for interpreting or explaining a model. simulation tests have been carried out before, but no study to date has isolated the effect of explanations on simulatability @xcite @xcite . we carry out simulation tests that are the first to incorporate all of the following design choices: (1) separating explained instances from test instances, so explanations do not give away the answers, (2) evaluating the effect of explanations against a baseline of unexplained examples, (3) balancing data by model correctness, so users cannot succeed by guessing the true label, and (4) forcing user predictions on all inputs, so performance is not biased toward overly specific explanations. we display our study design in figure 1 . we provide results from high-quality human we measure human users' ability to predict model behavior. we isolate the effect of explanations by first measuring baseline accuracy, then measuring accuracy after users are given access to explanations of model behavior. in the forward test, the explained examples are distinct from the test instances. in the counterfactual test, each test instance is a counterfactual version of a model input, and the explanations pertain to the original inputs. our key findings are: 1. lime improves forward and counterfactual simulatability in our tabular classification task. 2. prototype improves counterfactual simulatability across textual and tabular data domains. 3. no method definitively improves forward and counterfactual simulatability together on the text task, though our prototype and composite methods perform the best on average. 4. it appears that users' quality ratings of explanations are not predictive of how helpful the explanations are with counterfactual simulation. 5. while users rate composite explanations as among the best in quality, these combined explanations do not overtly improve simulatability in either data domain. ## explanation methods in this section, we describe the explanation methods. example explanations for a test movie review are shown in figure 2 . we limit our discussion of lime and anchor, since details for these methods can be found in the original papers. note that lime, anchor, and our decision boundary method can be used with arbitrary blackbox models. the prototype method is itself a neural model that also produces an explanation. @xcite present lime as a local linear approximation of model behavior. with a userspecified feature space, a linear model is fit to the blackbox outputs on samples from a distribution around an input. we set the number of features to use to 5, and we take class probabilities as our model output. when showing lime explanations to users, we give them the selected features with estimated weights, the model intercept, the sum of model weights, and the predicted model output. ## experimental design in this section, we describe our datasets, task models, user pool, and experimental design. ## results we report data from a total of 2166 responses from 39 user tests. each test is for a method and data domain pair, and contains either 16 or 32 task items, with some missingness due to users exiting the study early. in the results to follow, we use the term change to refer to our estimate of explanation effectiveness: the difference in user accuracy across prediction phases in simulation tests. we perform two-sided hypothesis tests for this quantity by a block bootstrap, resampling both users and unique task items within each condition @xcite . in addition, since users complete the first prediction round in either simulation test without access to explanations, we estimate the mean pre accuracy for each method with a random effects model. this allows us to share information across methods to yield more precise estimates of test performance. below, we analyze our experimental results and answer three questions: 1) do explanations help users? 2) how do users rate explanations? 3) can users predict explanation effectiveness? ## qualitative analysis when do explanations succeed at improving user accuracy, and when do they fail at doing so? below, we present example counterfactual test items, and we analyze how the explanations may have pointed to the reasons for model behavior. ## discussion forward tests stretch user memory. we show users 16 examples during learning phases but do not allow them to reference the learning data during prediction phases. reasonably, some users reported that it was difficult to retain insights from the learning phase during later prediction rounds. ## conclusion simulatability metrics give a quantitative measure of interpretability, capturing the intuition that explanations should improve a person's understanding of why a model produces its outputs. in this paper, we evaluated five explanation methods through simulation tests with text and tabular data. these are the first experiments to fully isolate the effect of algorithmic explanations on simulatability. we find clear improvements in simulatability only with lime for tabular data and our prototype method in counterfactual tests. it also appears that subjective user ratings of explanation quality are not predictive of explanation effectiveness in simulation tests. these results suggest that we must be careful about the metrics we use to evaluate explanation methods, and that there is significant room for improvement in current methods. randomly choose to make between 1 and 3 edits, then choose the features to edit uniformly at random, and finally pick new feature values uniformly at random. the only sampling constraint is that a variable cannot be set as its original value. for text data, we use a strategy that is similar to sampling from the perturbation distribution in @xcite , which is to randomly substitute words with their neighbors in glove word embedding space, sampling neighbors with probability proportional to their similarity. we make a few changes: we 1) decrease probability of token change with the length of sentence, 2) cap the number of edited words at 5 in the chosen perturbation if possible, and 3) limit edited tokens to be nouns, verbs, adjectives, adverbs, and adpositions. example perturbations are shown in the example of the user testing interface in figure 5 , which is given for a counterfactual test with text data.