What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods
Abstract
A multitude of explainability methods and associated fidelity performance metricshave been proposed to help better understand how modern AI systems make decisions.
However, much of the current work has remained theoretical – withoutmuch consideration for the human end-user.
In particular, it is not yet known(1) how useful current explainability methods are in practice for more real-worldscenarios and (2) how well associated performance metrics accurately predict howmuch knowledge individual explanations contribute to a human end-user tryingto understand the inner-workings of the system.
To fill this gap, we conductedpsychophysics experiments at scale to evaluate the ability of human participantsto leverage representative attribution methods for understanding the behavior ofdifferent image classifiers representing three real-world scenarios: identifying biasin an AI system, characterizing the visual strategy it uses for tasks that are toodifficult for an untrained non-expert human observer as well as understanding itsfailure cases.
Our results demonstrate that the degree to which individual attribution methods help human participants better understand an AI system variedwidely across these scenarios.
This suggests a critical need for the field to movepast quantitative improvements of current attribution methods towards the development of complementary approaches that provide qualitatively different sources ofinformation to human end-users.
Introduction
There is now broad consensus that modern AI systems might not be safe to be deployed in the realworld [1] despite their exhibiting very high levels of accuracy on held-out data because these systemshave been shown to exploit dataset biases and other statistical shortcuts [2–4].
A growing body ofresearch thus focuses on the development of explainability methods to better interpret these systems’predictions [5–12] to make them more trustworthy.
The application of these explainability methodswill find broad societal uses, like easing the debugging of self-driving vehicles [13] and helping tofulfill the “right to explanation” that European laws guarantee to its citizens [14].
The most commonly used methods in Explainable AI (XAI) are attribution methods, but despite aplethora of different approaches [5–9, 11, 15–17], assessing the quality and reliability of these methodsremains an open problem.
So far the community has mostly focused on evaluating these methodsusing surrogate measures defined axiomatically [13, 18–21].
The two most popular approaches are:(1) using ground truth annotations [8, 9, 11, 22–25] and (2) measuring faithfulness using objectivemetrics [8, 11, 22, 23, 26, 27].
The first approach takes humans into consideration only in so far as it evaluates the alignment of anexplanation with a human annotation.
This metric assumes that the classifier relies on human-likefeatures, which is an assumption that turns out to be erroneous [2–4], and unsurprisingly the results
As previously highlighted in [28], relying solely on current theoretical benchmarks to evaluateattribution methods can be a risky endeavor as they are totally detached from humans.
Indeed, theseminal work of [29] has emphasized keeping in mind the end goal of XAI: to develop usefulmethods that help the user better understand their model.
Thus, they recommend systematicallyhaving recourse to human experiences.
Moreover, a large body of research from the psychologyliterature has endorsed the argument that the functional role of explanations is to support learningand generalization [30–36] – i.e., can an explanation help identify generalizable rules that readilytransfer to unseen instances?
Therefore, evaluating attribution methods on their ability to help usersunderstand general rules to predict a classifier decisions is perfectly aligned with this vision.
To that end, [29] suggest evaluating explainability methods directly through the end goals of XAI.
We subscribe to this idea and, in this work, consider three main real-world scenarios that illustratepotential applications of explainability methods: (1) identifying a potential bias in a system’s decisionalong with its source using the classic Husky vs. Wolf dataset [5] (previous work has documentedthe risk associated with a biased model [37–39]); (2) identifying novel strategies discovered by asystem [40, 41] for tasks that would be nearly impossible for humans [42–45], here we consider ahighly complex categorization problem for non-expert humans [46, 47]; and (3) understanding failurecases [4, 37] on a subset of ImageNet [48].
The main contributions of this paper are as follows:
• We propose a novel human-centered explainability performance measure together with associatedpsychophysics methods to experimentally evaluate the practical usefulness of modern explainabilitymethods in real-world scenarios.
• Large-scale psychophysics experiments ( n = 1 , 150) revealed that SmoothGrad [7] is the mostuseful attribution method amongst all tested and that none of the fidelity performance metricsappear to predict if and when attribution methods will be of practical use to a human end-user.• Perceptual scores derived from attribution maps, characterizing either the complexity of an explanation or the challenge associated with identifying “what” features drive the system’s decision,appear to predict failure cases of explainability methods better than fidelity metrics.
Related work
Evaluations based on faithfulness measures Common approaches [8, 26] measure the fidelityof an explanation through the change in the classification score when the most important pixels areprogressively removed.
The bigger the drop, the more faithful is the explanation method.
To ensure that the score drop in score does not come from a change in the distribution of the perturbed images,the ROAR [49] methods include an additional step whereby the image classifier is re-trained betweeneach removal steps.
Because these methods do not require ground-truth annotations (i.e. object maskor bounding boxes), they are quite popular in computer vision [8, 11, 22, 23, 26, 27] and naturallanguage processing [11, 50, 51].
Nevertheless, faithfulness measures have recently been criticized as they all rely on a baselinefor removing important areas, a baseline that will obviously give better scores to methods relyinginternally on the same baseline.
More importantly, they do not consider humans at any time inthe evaluation.
As a result, it is unclear if the most faithful attribution method is practically useful tohumans.
Evaluations based on humans A second class of approaches consists in evaluating the ability ofhumans to leverage explanations for different purposes [5, 9, 28, 53–59].
[5] were the first to evaluatethe usefulness of explanations.
Their work focused on the use case of bias detection: they traineda classifier on a biased dataset of wolves and huskies and found the model consistently used thebackground to classify.
They asked participants if they trusted the model before and after seeing theexplanation for the model predictions.
They found that explanations helped detect bias in this case.
We use a similar dataset to reproduce those results as a verification, but our evaluation frameworkdiffers greatly from theirs as we do not ask if the participants trust the model but instead measuredirectly if they understand the model.
Closest to our work are [57–60].
The work described in [57–59] design their evaluation around thenotion of simulatability [29, 61].
They introduce different experimental procedures to measure ifhumans can learn from the explanations how to copy the model prediction on unseen data.
Someprovide the explanations at test time [57, 58].
Similar to us but for tabular data, [59] proposes to hideexplanation at test time, this forces the participants to learn the rules driving the model’s decision attraining time where the explanations are shown.
There are two limitations to their work: (1) theyprovide ground-truth label associated with the input image during training, (2) the participants seethe same set of images without explanation, and then with explanations, always in that order.
Thiscreates learning effect that can heavily bias their results.
We differ from their work by: (1) removingground-truth labels from our framework as they serve no purpose and can bias participants, and(2) we have different participants go through the different conditions.
This remove any learningeffect, and more importantly, new explainability methods can be evaluated independently and still becompared to the previously evaluated methods.
A very recent study [60] evaluated how AI systemsmay be able to assist human decisions by asking participants to identify the correct prediction out offour prediction-explanation pairs shown simultaneously.
This measure reflect how well explanationhelp users tell apart correct from incorrect predictions.
While the approach was useful to evaluateexplanation in this specific scenario, it is not clear how this framework could be used to evaluateexplainability methods more generally.
Furthermore, when comparing different types of methods,they adapt the complexity of certain explainability methods to ease the task for participants.
Weargue that the complexity of explanation is an important property of explanations and that abstractingit away from the evaluation lead to unfair comparison between methods.
In contrast, we propose amore general evaluation framework that can be used for any kind of explainability method withoutthe need to adapt them for the evaluation procedure – hence allowing for an unbiased comparisonbetween methods.
Proposed evaluation framework
Before providing a rigorous definition of interpretability, let us motivate our approach with anexample: a linear classifier is often considered to be readily interpretable because its inner workingis sufficiently intuitive that it can be comprehended by a human user.
A user can in turn build amental model of the classifier – potentially predicting the classifier’s output for arbitrary inputs.
Inessence, we suggest that the model is interpretable because the output can be predicted.
This conceptof predicting the classifier’s output is central to our approach and we conceptualize the human user asa
Meta-predictor of the machine learning model.
This notion of Meta-predictor is also closely relatedto the notion of simulatability [22, 29, 59, 61, 62].
We will now define the term more formally.
We consider a standard supervised learning setting where f is a black-box predictor that maps aninput x ∈ X (e.g., an image) to an output f ( x ) ∈ Y (e.g., a class label).
One of the main goalsof explainable AI is to yield useful rules to understand the inner-working of a model f such thatit is possible to infer its behavior on unseen data points.
To this end, a current approach consistsin studying explanations (Attribution Map, Concept Activation Vectors, Feature Visualization) for several predictions.
Formally, Φ is any explanation functional which, given a predictor f and a pointx , provides an information Φ ( f , x ) about the prediction of the predictor.
In our experiments, Φ isan attribution method but we would like to remind that the framework is naturally adaptable to otherexplainability methods such as concept-based methods or feature visualization.
The understandability-completeness trade-off Different attribution methods will typically produce different heatmaps – potentially highlighting different image regions and/or presenting the sameinformation in a different format.
The quality of an explanation can thus be affected by two factors:faithfulness of the explanation (i.e., how much pixels or input dimensions deemed important effectively drive the classifier’s prediction) and the understandability of the explanation for an end-user(i.e.
, how much of the pattern highlighted by the explanation is grasped by the user).
At one extreme, an explanation can be entirely faithful and provide all the information necessaryto predict how a classifier will assign a class label to an arbitrary image (by rendering all theparameters of the classifiers).
However, such information will obviously be too complex to beunderstood by a user and hence it is not understandable .
It should be noted that when the parametersare understandable, for a linear model, for example, it is generally the most suitable explanation.
Conversely, an explanation that overly simplifies the model might offer an approximation of the ruleused by the model that will be more easily grasped by the user –a more understandable explanation–but this approximation might ultimately mislead the user if it is not faithful .
That is to say, justbecause a human agrees with the evidence pointed out by an explanation does not necessarily meanthat it reflects how the model works.
Overall, this means that there is a trade-off between the amount of information provided by anexplanation and its comprehensibility to humans.
The most useful explanations lie somewhere in themiddle of this trade-off.
The usefulness metric We describe a new human-centered measure that incorporates this trade-offinto a single usefulness measure by empirically evaluating the ability of human participants to learnto “predict the predictor”, i.e., to be an accurate Meta-predictor.
Indeed, if an explanation allowsusers to infer precise rules for the functioning of the predictor on past data, the correct application ofthese same rules should allow the user to correctly anticipate the model’s decisions on future data.
Scrutable but inaccurate explanations will result in an inaccurate Meta-predictor – just like accurateinscrutable ones.
This Meta-predictor framework avoids current pitfalls such as confirmation bias- just because a user likes the explanation does not mean they will be a better Meta-predictor - orprediction leakage on the explanation - in simulatability experiments, as the explanation is availableduring the test phase, any explanation that leaks the prediction would have a perfect score, withoutgiving us any additional information about the model.
We will now formally describe the metric buildusing this framework.
We assume a dataset D
= { ( x i , f ( x i ) , Φ ( f , x i ) }
Ki =1 used to train human participants to learn topredict a classifier’s output f from K samples made of an input image x
i , the associated predictionsf ( x i ) and explanations Φ ( f , x i ) .
We denote ψ ( K ) a human Meta-predictor after being trained on the dataset D (see Fig. 2) using explanations.
In addition, let ψ (0) be the human Meta-predictor afterparticipants were trained on the same dataset but without explanations to offer baseline accuracyscores.
We can now define the usefulness of an explainability method Φ after training participantson K samples through the accuracy score of the Meta-predictor normalized by the baseline Metapredictor accuracy:
[Equation]
with P ( · ) the probability over a test set.
Thus, Utility- K score thus measures the improvement inaccuracy that the explanation has brought.
It is important to emphasize that this Utility measureonly depends on the classifier prediction and not on the ground-truth label as recommended by [63].
After fixing the number of training samples K , we compare the normalized accuracy of different Meta-predictor.
The Meta-predictor with the highest score is then the one whose explanations werethe most useful as measures compared to a no-explanation baseline.
Utility metric
In practice, we propose to vary the number of observations K ∈ { K 0 , ..., K n } andto consider an aggregated Utility score by computing the area under the curve (AUC) of the Utility- K .
The higher the AUC the better the corresponding explanation method is.
Formally, given a curverepresented by a set of n points C =
{ ( K 0 , Utility- K 0 ) , ..., ( K n , Utility- K n ) } where K i − 1 < K iwe define our main metric Utility = AUC ( C ) .
Experimental design
We first describe how participants were enrolled in the study, the exclusion criteria used to filter outuncooperative online participants, and our general experimental design (including both experimentaland control conditions).
Participants Behavioral accuracy data were gathered from n = 1 , 150 participants using the
Amazon Mechanical Turk (AMT) ( www.mturk.com ).
All participants provided informed consentelectronically and were compensated $1 . 4 for their time ( ∼ 5 − 8 min).The protocol was approvedby the University IRB and was carried out in accordance with the provisions of the World Medical
Association Declaration of Helsinki (See SI for details).
For each of the three tested datasets, weensured that there was a sufficient number of participants after filtering out uncooperative participants( n = 240 participants, 30 per condition, 8 conditions) to guarantee sufficient statistical power.
See SIfor details.
General study design It included 3 conditions: an experimental condition where an explanation isprovided to human participants during their training phase (see Fig. 2), a baseline condition whereno explanation was provided to the human participants, and a control condition where a bottom-upsaliency map [64] was provided as a non-informative explanation.
This last control is critical, andindeed lacking from previous work [5, 59], because it provides a control for the possibility thatproviding explanations along with training images simply increases participants’ engagement in thetask.
As we will show in Sec. 5, such non-informative explanation actually led to a decrease inparticipants’ ability to predict the classifier’s decisions – suggesting that giving a wrong explanationis worse than giving no explanation at all.
Each participant was only tested on a single condition to avoid possible experimental confounds.
Themain experiment was divided into 3 training sessions (with 5 training samples in each) each followedby a brief test.
In each individual trial, an image was presented with the associated prediction of themodel, either alone for the baseline condition or together with an explanation for the experimentaland control condition.
After a brief training phase (5 samples), participants’ ability to predict theclassifier’s output was evaluated on 7 new samples during a test phase.
During the test phase, noexplanation was provided to limit confounding effects: one possible effect is if the explanation leaksinformation about the class label.
* We also propose to use a reservoir that subjects can refer to duringthe testing phase to minimize memory load as a confounding factor which was reported in [59] (see
SI for an illustration).
Datasets and models We performed three distinct experiments in total – using a variety of neuralnetwork architectures and 6 representative attributions methods.
Each of these experiments aimed attesting the usefulness of the explanation in a different context
[Table]
methods that are statistically significant are bolded .
Our first scenario focuses on the detection of biases in AI systems using the popular Wolf vs. Huskydataset from ref.
[5] where an evaluation measure was already proposed around the usefulness ofexplanations for humans to detect biases.
This makes it a good control experiment to measure theeffectiveness of the framework proposed in Sec. 3.
For this first experiment, we used the same modelas in the original paper: InceptionV1 [66], and a similar dataset of Husky and Wolf images to biasthe model.
In this situation where prior knowledge of subjects can affect their Meta-predictor score,we balance data correctness ( 50% of correct/incorrect examples shown).
Therefore, a subject relyingonly on their prior knowledge will end up as a bad Meta-predictorof the model.
For this experiment,the results come from n = 242 subjects who all passed our screening process.
In our second scenario, we focus on a representative challenging image categorization task whichwould be hard to solve by a non-expert untrained human participant and the goal if for the end-userto understand the strategy that was discovered by the AI system.
Here, we chose the leaf datasetdescribed in [46].
We selected 2 classes from this dataset (Betulaceae and Celastracea) that couldnot be classified by shape to reduce the chances that participants will discover the solution on theirown – forcing them instead to rely on non-trivial features higlighted by the explanations (veins,leaf margin, etc).
This scenario is far from being artificial as it reflects a genuine problem for thepaleobotanist [47].
Can explainability methods help nonspecialists discover the strategies discoveredby an AI system?
As participants are lay people from Amazon Mechanical Turk we do not expectthem to be experts in botany but we did not explicitly tried to control for prior knowledge.
In thisexperiment, n = 240 subjects passed all our screening and other filtering processes.
Finally, our last scenario focuses on identifying cases where an AI system fails using ImageNet [48],also used in previous explainability work [11, 22, 28, 49, 57, 67].
We used this dataset because weexpect it to be representative of real-world scenarios where it is difficult to understand what the modelrelies on for classification which makes it very difficult to understand these failure cases.
Moreover,previous work has pointed out that attribution methods are not useful on this dataset [57], we havethus chosen to extend our analysis to this particular case.
We use a ResNet50 [68] pretrained on thisdataset as predictor.
Because prior knowledge is a major confounding factor on ImageNet, we selecta pair of classes that was heavily miss-classified by the model, to be able to show subjects 50% ofcorrect/incorrect predictions: the pair Kit Fox and Red Fox fits this requirement.
In this experiment,we analyzed data from n = 241 participants who passed our screening and filtering processes.
For all experiments, we compared 6 representative attribution methods: Saliency (SA) [15], Gradient⊙ Input (GI) [17], Integrated Gradients (IG) [6], Occlusion (OC) [16], SmoothGrad (SG) [7] and
Grad-CAM (GC) [9].
Further information on these methods can be found in SI.
Table 1 summarizesall the results from our psychophysics experiments.
Results
Scenario (1): Bias detection Fig.
3 shows the Utility- K scores for each method after differentnumbers of training samples were used to train participants for the biased dataset of Husky vs. Wolf.
The Utility score encodes the quality of the explanations provided by a method, the higher the score,the better the method, with the baseline score being 1 (every score is divided by the baseline scorecorresponding to human accuracy after training without explanations).
A first observation is that the explanations have a positive effect on the Utility- K score: the explanationallows participants to better predict the model’s decision (as the Utility scores are above 1).
These results are consistent with those reported in [5].
This is confirmed with an Analysis of Variance(ANOVA) for which we found a significant main effect, with a medium effect size ( F (7 , 234) =9 .
19 , p < .
001 , η 2 = 0 . 082 ).
Moreover, the only score below the baseline is that of the controlexplanation, which do not make use of the model.
We further explore our results by performingpairwise comparisons using Tukey’s Honestly Significant Difference [69] to compare the differentexplanations against the baseline.
We found 3 explainability methods to be significantly better thanthe baseline: Grad-CAM( p < 0 .
001 ), Occlusion( p = 0 . 01 ) and SmoothGrad( p = 0 . 034 ).
Thus,participants who received the Grad-CAM, Occlusion or SmoothGrad explanations performed much better than those who did not receive them.
Scenario (2):
Identifying an expert strategy In Fig.
3, we show results onthe Leaves dataset.
An ANOVA analysis across all conditions revealed a significant main effect, albeit small ( F (7 , 232) [Equation] [Equation]
[Equation]
The results suggest that current faithfulness metrics are poor predictors of the end-goal usefulness of explanation methods.
Concerning the ImageNet dataset (triangle marker), the Utility scores are insignificant since none of the methods improves the baseline.
This implies that explanation also had a positive effect resulting in better Meta-predictor in this use case.
A Tukey’s Honestly Significant Difference test suggests that the best explanations are Saliency , SmoothGrad and Integrated Gradients as they are the onlyones to be significantly better than our baseline (WE)( p = .
004 , p = .
007 and p = .
03 respectively).
An interesting result is that SmoothGrad seems tobe consistently useful across both use cases whereexplanations are indeed practically useful.
A moresurprising result is that Saliency which was one ofthe worst explanations for bias detection, is now thebest explanation on this use case.
Scenario (3): Understanding failure cases Table 1 shows that, on the ImageNet dataset, none ofthe methods tested exceeded baseline accuracy.
Indeed, the experiment carried out, even with an improved experimental design, led us to the same conclusion as previous works [57]: none of the tested attribution methods are useful (ANOVA: F (7 , 233)
=1 . 26 , p > . 05 ).
In the use case of understanding failure cases on ImageNet, no attributions methodsseem to be useful.
Why do attribution methods fail?
After studying the usefulness of attribution methods across 3 real-world scenarios for explainable AIwe found that attribution methods help in some cases but not others.
To try to better understand why this is the case, we explore 3 possible explanations to gain insights as to why sometimes attributionmethods fail to help.
Faithfulness as a proxy for Utility?
Faithfulness is often described as one of the keydesiderata for a good explanation [21, 70, 71].
If an explanation fails to be sufficiently faithful, the rules it highlights won’t allow a user to understand the inner-workingof the model.
Thus, a lack of faithfulness on ImageNet could explain our results.
This suggest a weak but possibly existing link between Complexity and Utility scores.
To test this hypothesis, we use the faithfulness metrics: Deletion[8, 26], commonly used to compareattribution methods [8, 11, 22, 23, 26, 27].
A low
Deletion score indicates a good faithfulness , thus forease of reading we report the faithfulness score as1 − Deletion such that a higher faithfulness score isbetter.
Fig 4 shows the linear relationship between our Utility metrics and the faithfulness scores computed forevery attribution method across all 3 datasets.
We observe two main trends: 1) There does not appear tobe any specific pattern regarding faithfulness thatcould explain why attribution methods are not useful for ImageNet , and 2) the least useful attributionmethods for both use cases for which methods help(Bias and Leaves) are some of the leading methodsin the field measured by the faithfulness metric ( ?? ).
We also found a weak, if maybe anti-correlated,between faithfulness and usefulness: just focusing on making attribution methods more faithful doesnot translate to having methods with higher practical usefulness for end-users.
And, in fact, focusingtoo heavily on faithfulness seem to come at the expense of usefulness, resulting in explanations thatare counter-intuitively less useful.
This second observation may seem rather alarming for the fieldgiven that the faithfulness measure is one of the main benchmark.
The perceptual similarity of highlighted regions by a given attribution method for both classes is measured, for each method, for each dataset.
We observe a strong correlation between the perceptual similarity of features highlighted for both classes and the practical usefulness of methods.
Are explanations too complex?
Using thetrade-off between completeness and understandability previously discussed in section 3, we formulate another plausible hypothesis: some explanations may be faithful but too complex andtherefore not understood by humans.
In thatview, an explanation with low complexity wouldtend to be more useful.
As a simple measure of the complexity of visualexplanations, it would be ideal to be able to compute the Kolmogorov complexity [72] of eachexplanation.
It was shown in previous work [73]to correlate well with human-derived ratings forthe complexity of natural images [74, 75].
Assuggested by [72, 76] we used a standard compression techniques (JPEG) to approximate the
Kolmogorov complexity.
Fig.
5 shows the Utility vs complexity score of attribution methodsfor each dataset.
For one of the datasets whereattribution methods help the results suggest the presence of a strong correlation between usefulnessand complexity : the least complex method is the most useful to end-users.
For the other datasets, theresults are either not conclusive (Leaves), or are not relevant as method are not useful (ImageNet).
Furthermore, across datasets there is no significant difference in the complexity of explanationsthat can explain why attribution methods do not help on ImageNet.
This could be because the
Kolmogorov Complexity does not perfectly reflects human visual complexity, or because this is notthe key element to explain failure cases of attribution methods.
An intrinsic limitation of Attribution methods?
The role of attribution methods is to help identify“where” to look in an image to understand the basis for a system’s prediction.
However, attributionmethods do not tell us “what” in those image regions is driving decisions.
Understanding thewhat information is thus left open for the end-user to infer.
For visually very similar classes (such as discriminating between different breeds of dogs) and fine-grained recognition more generally,understanding what precise features are being used by the system might be very challenging forhuman simply by looking at those diagnostic image regions (knowing that the system is looking at theear regions tells us nothing about what in those ear regions is driving decisions).
Our main hypothesisis that such a lack of clear “what” understanding is precisely what is driving the failure of attributionmethods on the ImageNet dataset.
To test this hypothesis we estimated the perceptual similaritybetween highlighted regions of both classes, across methods and across datasets (more results in SI).
If the perceptual similarity is high for two distinct classification we except attribution to not be useful.
To measure perceptual similarity we use the LPIPS metric [77] as it has been shown to provide a goodproxy for human perceptual similarity judgments [77, 78].
We report the perceptual similarity scoreas 1 - LPIPS score so that a high score means a high similarity.
Fig.
6 shows the correlation betweenthe perceptual similarity scores vs. our Utility scores on all methods and datasets studied.
Ourresults suggest a strong correlation between perceptual similarity and practical usefulness: the moreperceptually similar discriminative features of both class are, the less useful the methods become.
More importantly, the results across dataset show that on ImageNet, where attribution methods donot help, every methods has a high similarity score.
This result suggest that after a certain thresholdof perceptual similarity, attribution methods might no longer be useful, no matter how faithful orlow in complexity the explanation is.
Overall, the results suggest that the perceptual similarity ofdiscriminative features could explain why attribution methods fail on ImageNet.
Discussion
In summary, we conducted a large-scale human psychophysics experiment to test the utility ofexplainability methods in real-world scenarios.
Our work shows that in two of the three testedscenarios (bias detection and identification of new strategies), explainability has indeed progressedand provided meaningful assistance to human end-users.
Nevertheless, we identified a scenario(understanding failure case) for which none of the tested attribution methods were useful.
This resultis consistent with previous work [57] and highlights a fundamental challenge for XAI.
Further analysis of associated fidelity performance metrics used to guide the development of explainability methods revealed that they did not correlate with our empirical measure of utility – suggestingthat they fail to predict such failure cases.
We also investigated the possibility that the complexityof individual explanations may play a role in explaining human failures to learn to leverage thoseexplanations to understand the model and, while we found a weak correlation between complexityand our empirical measure of utility, this correlation appears too low to explain these failure cases.
Finally, we considered the possibility that fully understanding the inner-workings of a vision systemrequires understanding not just “where” to look (as revealed by attribution maps) but also “what”features in those locations drive decisions, something not currently revealed by these methods.
Our assumption is that the need for finer “what” information should arise when diagnostic imagelocations across classes look perceptually very similar and potentially semantically related for certainclassification problems (e.g., looking at the ears or the snout to discriminate between breeds of catsand dogs) and one need to identify what aspects of these features is driving decisions.
We computed aperceptual score for classification problems by estimating the perceptual similarity between diagnosticimage regions (as predicted by attribution methods) and found that, indeed, when this score predicts acertain level of perceptual similarity between two distinct classes, attribution methods fail to contributeuseful information to human end-users, regardless of the fidelity or complexity of the explanations.
This fundamental result highlights an intrinsic limitation of attribution methods and suggests the needfor a change of focus in the field – from quantitative improvements of current attribution methods tothe development of novel explainability methods that communicate qualitatively different knowledgeabout the inner-workings of the system.
The cost of evaluating additional methods using our benchmark is relatively modest ($50 per test scenario).
By making all data and software available, we hope to help researchers evaluate future explainability methods and assess overall progress towards the development of more human-interpretable AIsystems.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?
[Yes] The limit of a human evaluation is often the cost, which we address in sec (c) Did you discuss any potential negative societal impacts of your work? [No] We identify
no potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] In the SI.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] in sec 4 and in SI.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
in SI.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[Yes] Screenshot of the experiments are in the SI(b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes] Risk are specified in the consent form inSI.(c)
Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
{+[Yes] in sec 
+}