What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods
Abstract
A multitude of explainability methods and associated fidelity performance metricshave been proposed to help better understand how modern AI systems make decisions.
However, much of the current work has remained theoretical – withoutmuch consideration for the human end-user.
In particular, it is not yet known(1) how useful current explainability methods are in practice for more real-worldscenarios and (2) how well associated performance metrics accurately predict howmuch knowledge individual explanations contribute to a human end-user tryingto understand the inner-workings of the system.
To fill this gap, we conductedpsychophysics experiments at scale to evaluate the ability of human participantsto leverage representative attribution methods for understanding the behavior ofdifferent image classifiers representing three real-world scenarios: identifying bias in an AI system, characterizing the visual strategy it uses for tasks that are toodifficult for an untrained non-expert human observer as well as understanding itsfailure cases.
Our results demonstrate that the degree to which individual attribution methods help human participants better understand an AI system variedwidely across these scenarios.
This suggests a critical need for the field to movepast quantitative improvements of current attribution methods towards the development of complementary approaches that provide qualitatively different sources ofinformation to human end-users.
Introduction
There is now broad consensus that modern AI systems might not be safe to be deployed in the realworld [1] despite their exhibiting very high levels of accuracy on held-out data because these systemshave been shown to exploit dataset biases and other statistical shortcuts [2–4].
A growing body ofresearch thus focuses on the development of explainability methods to better interpret these systems’predictions [5–12] to make them more trustworthy.
The application of these explainability methodswill find broad societal uses, like easing the debugging of self-driving vehicles [13] and helping tofulfill the “right to explanation” that European laws guarantee to its citizens [14].
The most commonly used methods in Explainable AI (XAI) are attribution methods, but despite aplethora of different approaches [5–9, 11, 15–17], assessing the quality and reliability of these methodsremains an open problem.
So far the community has mostly focused on evaluating these methodsusing surrogate measures defined axiomatically [13, 18–21].
The two most popular approaches are:(1) using ground truth annotations [8, 9, 11, 22–25] and (2) measuring faithfulness using objectivemetrics [8, 11, 22, 23, 26, 27].
The first approach takes humans into consideration only in so far as it evaluates the alignment ofan explanation with a human annotation.
This metric assumes that the classifier relies on human-
As previously highlighted in [28] relying solely on current theoretical benchmarks to evaluateattribution methods can be a risky endeavor as they are totally detached from humans.
Indeed, theseminal work of [29] has emphasized keeping in mind the end goal of XAI: to develop usefulmethods that help the user better understand their model.
Thus they recommend systematicallyhaving recourse to human experiences.
Moreover, a large body of research from the psychologyliterature has endorsed the argument that the functional role of explanations is to support learningand generalization [30–36] – i.e., can an explanation help identify generalizable rules that readilytransfer to unseen instances?
Therefore, evaluating attribution methods on their ability to help usersunderstand general rules to predict a classifier decisions is perfectly aligned with this vision.
To that end, [29] suggest evaluating explainability methods directly through the end goals of XAI.
We subscribe to this idea and, in this work, consider three main real-world scenarios that illustratepotential applications of explainability methods: (1) identifying a potential bias in a system’s decisionalong with its source using the classic Husky vs. Wolf dataset [5] (previous work has documentedthe risk associated with a biased model [37–39]); (2) identifying novel strategies discovered by asystem [40, 41] for tasks that would be nearly impossible for humans [42–45], here we consider ahighly complex categorization problem for non-expert humans [46, 47]; and (3) understanding failurecases [4, 37] on a subset of ImageNet [48].
The main contributions of this paper are as follows:
• We propose a novel human-centered explainability performance measure together with associatedpsychophysics methods to experimentally evaluate the practical usefulness of modern explainabilitymethods in real-world scenarios.
• Large-scale psychophysics experiments ( n = 1 , 150) revealed that SmoothGrad [7] is the mostuseful attribution method amongst all tested and that none of the fidelity performance metrics appear to predict if and when attribution methods will be of practical use to a human end-user. • Perceptual scores derived from attribution maps, characterizing either the complexity of an expla- nation or the challenge associated with identifying “what” features drive the system’s decision,
appear to predict failure cases of explainability methods better than fidelity metrics.
Related work
Evaluations based on faithfulness measures Common approaches [8, 26] measure the fidelityof an explanation through the change in the classification score when the most important pixels areprogressively removed.
The bigger the drop, the more faithful is the explanation method.
To ensurethat the drop in score does not come from a change in the distribution of the perturbed images, the ROAR [49] methods include an additional step whereby the image classifier is re-trained betweeneach removal step.
Because these methods do not require ground-truth annotations (i.e. object masksor bounding boxes), they are quite popular in computer vision [8, 11, 22, 23, 26, 27] and naturallanguage processing [11, 50, 51].
Nevertheless, faithfulness measures have recently been criticized as they all rely on a baselinefor removing important areas, a baseline that will obviously give better scores to methods relyinginternally on the same baseline [52].
More importantly, they do not consider humans at any time inthe evaluation.
As a result, it is unclear if the most faithful attribution method is practically useful tohumans.
Evaluations based on humans A second class of approaches consists in evaluating the ability ofhumans to leverage explanations for different purposes [5, 9, 28, 53–59].
[5] were the first to evaluatethe usefulness of explanations.
Their work focused on the use case of bias detection: they traineda classifier on a biased dataset of wolves and huskies and found the model consistently used thebackground to classify.
They asked participants if they trusted the model before and after seeing theexplanation for the model predictions.
They found that explanations helped detect bias in this case.
We use a similar dataset to reproduce those results as a verification, but our evaluation frameworkdiffers greatly from theirs as we do not ask if the participants trust the model but instead measuredirectly if they understand the model.
Closest to our work are [57–60].
[57–59] design their evaluation around the notion of simulatability [29, 61].
They introduce different experimental procedures to measure if humans can learn fromthe explanations how to copy the model prediction on unseen data.
Some provide the explanationsat test time [57, 58].
Similar to us but for tabular data, [59] proposes to hide explanations at testtime, this forces the participants to learn the rules driving the model’s decision at training time wherethe explanations are shown.
There are two limitations to their work: (1) they provide ground-truthlabels associated with input images during training, (2) the participants see the same set of imageswithout explanation, and then with explanations, always in that order.
This creates learning effectsthat can heavily bias their results.
We differ from their work by: (1) removing ground-truth labelsfrom our framework as they serve no purpose and can bias participants, and (2) we have different participants go through the different conditions.
This removes any learning effect, and more importantly,new explainability methods can be evaluated independently and still be compared to the previouslyevaluated methods.
A very recent study [60] evaluated how AI systems may be able to assist humandecisions by asking participants to identify the correct prediction out of four prediction-explanationpairs shown simultaneously.
This measure reflects how well explanations help users tell apart correctfrom incorrect predictions.
While the approach was useful to evaluate explanations in this specificscenario, it is not clear how this framework could be used to evaluate explainability methods moregenerally.
Furthermore, when comparing different types of methods, they adapt the complexity ofcertain explainability methods to ease the task for participants.
We argue that the complexity ofexplanation is an important property of explanations and that abstracting it away from the evaluationlead to unfair comparisons between methods.
In contrast, we propose a more general evaluationframework that can be used for any kind of explainability method without the need to adapt them forthe evaluation procedure – hence allowing for an unbiased comparison between methods.
Proposed evaluation framework
Before providing a rigorous definition of interpretability, let us motivate our approach with anexample: a linear classifier is often considered to be readily interpretable because its inner workingis sufficiently intuitive that it can be comprehended by a human user.
A user can in turn build amental model of the classifier – potentially predicting the classifier’s output for arbitrary inputs.
Inessence, we suggest that the model is interpretable because the output can be predicted.
This conceptof predicting the classifier’s output is central to our approach and we conceptualize the human user asa
Meta-predictor of the machine learning model.
This notion of Meta-predictor is also closely relatedto the notion of simulatability [22, 29, 59, 61, 62].
We will now define the term more formally.
We consider a standard supervised learning setting where f is a black-box predictor that maps aninput x ∈ X (e.g., an image) to an output f ( x ) ∈ Y (e.g., a class label).
One of the main goals of explainable AI is to yield useful rules to understand the inner-working of a model f such thatit is possible to infer its behavior on unseen data points.
To this end, a current approach consistsin studying explanations (Attribution Map, Concept Activation Vectors, Feature Visualization) forseveral predictions.
Formally, Φ is any explanation functional which, given a predictor f and a pointx , provides an information Φ ( f , x ) about the prediction of the predictor.
In our experiments, Φ isan attribution method but we would like to remind that the framework is naturally adaptable to otherexplainability methods such as concept-based methods or feature visualization.
The understandability-completeness trade-off Different attribution methods will typically produce different heatmaps – potentially highlighting different image regions and/or presenting the sameinformation in a different format.
The quality of an explanation can thus be affected by two factors:faithfulness of the explanation (i.e., how many pixels or input dimensions deemed important effectively drive the classifier’s prediction) and the understandability of the explanation for an end-user(i.e.
, how much of the pattern highlighted by the explanation is grasped by the user).
At one extreme, an explanation can be entirely faithful and provide all the information necessaryto predict how a classifier will assign a class label to an arbitrary image (by rendering all theparameters of the classifiers).
However, such information will obviously be too complex to beunderstood by a user and hence it is not understandable .
It should be noted that when the parametersare understandable, for a linear model, for example, it is generally the most suitable explanation.
Conversely, an explanation that overly simplifies the model might offer an approximation of the ruleused by the model that will be more easily grasped by the user –a more understandable explanation–but this approximation might ultimately mislead the user if it is not faithful .
That is to say, justbecause a human agrees with the evidence pointed out by an explanation does not necessarily meanthat it reflects how the model works.
Overall, this means that there is a trade-off between the amount of information provided by anexplanation and its comprehensibility to humans.
The most useful explanations lie somewhere in themiddle of this trade-off.
The usefulness metric We describe a new human-centered measure that incorporates this trade-offinto a single usefulness measure by empirically evaluating the ability of human participants to learnto “predict the predictor”, i.e., to be an accurate Meta-predictor.
Indeed, if an explanation allowsusers to infer precise rules for the functioning of the predictor on past data, the correct application ofthese same rules should allow the user to correctly anticipate the model’s decisions on future data.
Scrutable but inaccurate explanations will result in an inaccurate Meta-predictor – just like accurateinscrutable ones.
This Meta-predictor framework avoids current pitfalls such as confirmation bias- just because a user likes the explanation does not mean they will be a better Meta-predictor - orprediction leakage on the explanation - in simulatability experiments, as the explanation is availableduring the test phase, any explanation that leaks the prediction would have a perfect score, withoutgiving us any additional information about the model.
We will now formally describe the metric buildusing this framework.
We assume a dataset D
= { ( x i , f ( x i ) , Φ ( f , x i ) }
Ki =1 used to train human participants to learn topredict a classifier’s output f from K samples made of an input image x
i , the associated predictionsf ( x i ) and explanations Φ ( f , x i ) .
We denote ψ ( K ) a human Meta-predictor after being trained onthe dataset D (see Fig. 2) using explanations.
In addition, let ψ (0) be the human Meta-predictor afterparticipants were trained on the same dataset but without explanations to offer baseline accuracyscores.
We can now define the usefulness of an explainability method Φ after training participantson K samples through the accuracy score of the Meta-predictor normalized by the baseline Metapredictor accuracy:
[Equation]
with P ( · ) the probability over a test set.
Thus, Utility- K score measures the improvement in accuracythat the explanation has brought.
It is important to emphasize that this Utility measure only dependson the classifier prediction and not on the ground-truth label as recommended by [63].
After fixingthe number of training samples K , we compare the normalized accuracy of different Meta-predictor.
The Meta-predictor with the highest score is then the one whose explanations were the most useful asmeasures compared to a no-explanation baseline.
Utility metric
In practice, we propose to vary the number of observations K ∈ { K 0 , ..., K n } andto consider an aggregated Utility score by computing the area under the curve (AUC) of the Utility- K .
The higher the AUC the better the corresponding explanation method is.
Formally, given a curverepresented by a set of n points C =
{ ( K 0 , Utility- K 0 ) , ..., ( K n , Utility- K n ) } where K i − 1 < K iwe define our main metric Utility = AUC ( C ) .
Experimental design
We first describe how participants were enrolled in the study, then our general experimental design(See SI for more informations).
Participants Behavioral accuracy data were gathered from n = 1 , 150 participants using the
Amazon Mechanical Turk (AMT) ( www.mturk.com ).
All participants provided informed consentelectronically and were compensated $1 . 4 for their time ( ∼ 5 − 8 min). The protocol was approvedby the University IRB and was carried out in accordance with the provisions of the World Medical
Association Declaration of Helsinki (See SI for details).
For each of the three tested datasets, weensured that there was a sufficient number of participants after filtering out uncooperative participants( n = 240 participants, 30 per condition, 8 conditions) to guarantee sufficient statistical power.
See SIfor details.
General study design It included 3 conditions: an experimental condition where an explanation isprovided to human participants during their training phase (see Fig. 2), a baseline condition whereno explanation was provided to the human participants, and a control condition where a bottom-upsaliency map [64] was provided as a non-informative explanation.
This last control is critical, andindeed lacking from previous work [5, 59], because it provides a control for the possibility thatproviding explanations along with training images simply increases participants’ engagement in thetask.
As we will show in Sec. 5, such non-informative explanations actually led to a decrease inparticipants’ ability to predict the classifier’s decisions – suggesting that giving a wrong explanationis worse than giving no explanations at all.
Each participant was only tested on a single condition to avoid possible experimental confounds.
Themain experiment was divided into 3 training sessions (with 5 training samples in each) each followedby a brief test.
In each individual trial, an image was presented with the associated prediction of themodel, either alone for the baseline condition or together with an explanation for the experimentaland control condition.
After a brief training phase (5 samples), participants’ ability to predict theclassifier’s output was evaluated on 7 new samples during a test phase.
During the test phase, noexplanation was provided to limit confounding effects: one possible effect is if the explanation leaksinformation about the class label.
* We also propose to use a reservoir that subjects can refer to duringthe testing phase to minimize memory load as a confounding factor which was reported in [59] (see
SI for an illustration).
[Table]
methods that are statistically significant are bolded .
Datasets and models We performed three distinct experiments in total – using a variety of neuralnetwork architectures and 6 representative attributions methods.
Each of these experiments aimed attesting the usefulness of the explanation in a different context.
Our first scenario focuses on the detection of biases in AI systems using the popular Wolf vs.
Husky dataset from [5] where an evaluation measure was already proposed around the usefulness ofexplanations for humans to detect biases.
This makes it a good control experiment to measure theeffectiveness of the framework proposed in Sec. 3.
For this first experiment, we used the same modelas in the original paper: InceptionV1 [66], and a similar dataset of Husky and Wolf images to biasthe model.
In this situation where prior knowledge of subjects can affect their Meta-predictor score,we balance data correctness ( 50% of correct/incorrect examples shown).
Therefore, a subject relyingonly on their prior knowledge will end up as a bad Meta-predictorof the model.
For this experiment,the results come from n = 242 subjects who all passed our screening process.
In our second scenario, we focus on a representative challenging image categorization task whichwould be hard to solve by a non-expert untrained human participant and the goal is for the end-userto understand the strategy that was discovered by the AI system.
Here, we chose the leaf datasetdescribed in [46].
We selected 2 classes from this dataset (Betulaceae and Celastracea) that couldnot be classified by shape to reduce the chances that participants will discover the solution on theirown – forcing them instead to rely on non-trivial features highlighted by the explanations (veins,leaf margin, etc).
This scenario is far from being artificial as it reflects a genuine problem for thepaleobotanist [47].
Can explainability methods help nonspecialists discover the strategies discoveredby an AI system?
As participants are lay people from Amazon Mechanical Turk we do not expectthem to be experts in botany, therefore we did not explicitly try to control for prior knowledge.
In thisexperiment, n = 240 subjects passed all our screening and other filtering processes.
Finally, our last scenario focuses on identifying cases where an AI system fails using ImageNet [48],also used in previous explainability work [11, 22, 28, 49, 57, 67].
We used this dataset because weexpect it to be representative of real-world scenarios where it is difficult to understand what the modelrelies on for classification which makes it very difficult to understand these failure cases.
Moreover,previous work has pointed out that attribution methods are not useful on this dataset [57], we havethus chosen to extend our analysis to this particular case.
We use a ResNet50 [68] pretrained on thisdataset as predictor.
Because prior knowledge is a major confounding factor on ImageNet, we selecta pair of classes that was heavily miss-classified by the model, to be able to show subjects 50% ofcorrect/incorrect predictions: the pair Kit Fox and Red Fox fits this requirement.
In this experiment,we analyzed data from n = 241 participants who passed our screening and filtering processes.
For all experiments, we compared 6 representative attribution methods: Saliency (SA) [15], Gradient⊙ Input (GI) [17], Integrated Gradients (IG) [6], Occlusion (OC) [16], SmoothGrad (SG) [7] and
Grad-CAM (GC) [9].
Further information on these methods can be found in SI.
Table 1 summarizesall the results from our psychophysics experiments.
Results
Scenario (1): Bias detection Fig.
3 shows the Utility- K scores for each method after differentnumbers of training samples were used to train participants for the biased dataset of Husky vs. Wolf.
The Utility score encodes the quality of the explanations provided by a method, the higher the score, the better the method, with the baseline score being 1 (every score is divided by the baseline score corresponding to accuracy
A first observation is that the explanations have a positive effect on the Utility- K score: the explanationallows participants to better predict the model’s decision (as the Utility scores are above 1).
Theseresults are consistent with those reported in [5].
This is confirmed with an Analysis of Variance(ANOVA) for which we found a significant main effect, with a medium effect size ( F (7 , 234) =9 .
19 , p < .
001 , η 2 = 0 . 082 ).
Moreover, the only score below the baseline is that of the controlexplanation, which do not make use of the model.
We further explore our results by performingpairwise comparisons using Tukey’s Honestly Significant Difference [69] to compare the differentexplanations against the baseline.
We found 3 explainability methods to be significantly better thanthe baseline: Grad-CAM( p < 0 .
001 ), Occlusion( p = 0 . 01 ) and SmoothGrad( p = 0 . 034 ).
Thus,participants who received the Grad-CAM, Occlusion or SmoothGrad explanations performed muchbetter than those who did not receive them.
Scenario (2):
Identifying an expert strategy In Fig.
3, we show results onthe Leaves dataset.
An ANOVA analysis across all conditions revealed a significant main effect, albeit small ( F (7 , 232)
[Equation]
The results suggest that current faithfulness metrics are poor predictors of the end-goal usefulness of explanation methods.
Concerning the ImageNet dataset (triangle marker), the Utility scores are insignificant since none of the methods improves the baseline.
This implies that explanation also had a positive effect resulting in better Meta-predictor in this use case.
A Tukey’s Honestly Significant Difference test suggests that the best explanations are Saliency , SmoothGrad and Integrated Gradients as they are the onlyones to be significantly better than our baseline (WE)( p = .
004 , p = .
007 and p = .
03 respectively).
An interesting result is that SmoothGrad seems tobe consistently useful across both use cases whereexplanations are indeed practically useful.
A moresurprising result is that Saliency which was one ofthe worst explanations for bias detection, is now thebest explanation on this use case.
Scenario (3): Understanding failure cases Table 1 shows that, on the ImageNet dataset, none ofthe methods tested exceeded baseline accuracy.
Indeed, the experiment carried out, even with an improved experimental design, led us to the same conclusion as previous works [57]: none of the tested attribution methods are useful (ANOVA: F (7 , 233)
=1 . 26 , p > . 05 ).
In the use case of understanding failure cases on ImageNet, no attributions methodsseem to be useful.
Why do attribution methods fail
After studying the usefulness of attribution methods across 3 real-world scenarios for explainable AIwe found that attribution methods help in some cases but not others.
To try to better understand whythis is the case, we explore 3 possible explanations to gain insights as to why sometimes attributionmethods fail to help.
Faithfulness as a proxy for Utility?
Faithfulness is often described as one of the keydesiderata for a good explanation [21, 70, 71].
If an explanation fails to be sufficiently faithful, the rules it highlights won’t allow a user to understand the inner-workingof the model.
Thus, a lack of faithfulness on ImageNet could explain our results.
This suggest a weak but possibly existing link between Complexity and Utility scores.
To test this hypothesis, we use the faithfulness metrics: Deletion[8, 26], commonly used to compareattribution methods [8, 11, 22, 23, 26, 27].
A low
Deletion score indicates a good faithfulness , thus forease of reading we report the faithfulness score as1 − Deletion such that a higher faithfulness score isbetter.
Fig 4 shows the linear relationship between our Utility metrics and the faithfulness scores computed forevery attribution method across all 3 datasets.
Weobserve two main trends: 1) There does not appearto be any specific pattern regarding faithfulnessthat could explain why attribution methods arenot useful for ImageNet , and 2) the least useful attribution methods for both use cases for which methodshelp (Bias and Leaves) are some of the leading methods in the field measured by the faithfulness metric.
We also found a weak, if maybe anti-correlated,relation between faithfulness and usefulness: just focusing on making attribution methods morefaithful does not translate to having methods with higher practical usefulness for end-users.
And,in fact, focusing too heavily on faithfulness seem to come at the expense of usefulness, resultingin explanations that are counter-intuitively less useful.
This second observation may seems ratheralarming for the field given that the faithfulness measure is one of the main benchmarks.
The perceptual similarity of highlighted regions by a given attribution method for both classes is measured, for each method, for each dataset.
We observe a strong correlation between the perceptual similarity of features highlighted for both classes and the practical usefulness of methods.
Are explanations too complex?
Using thetrade-off between completeness and understandability previously discussed in section 3, we formulate another plausible hypothesis: some explanations may be faithful but too complex andtherefore not understood by humans.
In thatview, an explanation with low complexity wouldtend to be more useful.
As a simple measure of the complexity of visualexplanations, it would be ideal to be able to compute the Kolmogorov complexity [72] of eachexplanation.
It was shown in previous work [73]to correlate well with human-derived ratings forthe complexity of natural images [74, 75].
Assuggested by [72, 76] we used a standard compression technique (JPEG) to approximate the
Kolmogorov complexity.
Fig.
5 shows the Utility vs complexity score of attribution methodsfor each dataset.
For one of the datasets whereattribution methods help the results suggest the presence of a strong correlation between usefulnessand complexity : the least complex method is the most useful to end-users.
For the other datasets, theresults are either not conclusive (Leaves), or are not relevant as methods are not useful (ImageNet).
Furthermore, across datasets there is no significant difference in the complexity of explanationsthat can explain why attribution methods do not help on ImageNet.
This could be because the
Kolmogorov Complexity does not perfectly reflect human visual complexity, or because this is notthe key element to explain failure cases of attribution methods.
An intrinsic limitation of Attribution methods?
The role of attribution methods is to help identify“where” to look in an image to understand the basis for a system’s prediction.
However, attributionmethods do not tell us “what” in those image regions is driving decisions.
Understanding thewhat information is thus left open for the end-user to infer.
For visually very similar classes (suchas discriminating between different breeds of dogs) and fine-grained recognition more generally,understanding what precise features are being used by the system might be very challenging forhumans simply by looking at those diagnostic image regions (knowing that the system is lookingat the ear regions tells us nothing about what in those ear regions is driving decisions).
Our mainhypothesis is that such a lack of clear “what” understanding is precisely what is driving the failureof attribution methods on the ImageNet dataset.
To test this hypothesis we estimated the perceptualsimilarity between highlighted regions of both classes, across methods, and across datasets (moreresults in SI).
If the perceptual similarity is high for two distinct classifications we expect attributionto not be useful.
To measure perceptual similarity we use the LPIPS metric [77] as it has beenshown to provide a good proxy for human perceptual similarity judgments [77, 78].
We report theperceptual similarity score as 1 - LPIPS score so that a high score means a high similarity.
Fig.
6shows the correlation between the perceptual similarity scores vs. our Utility scores on all methodsand datasets studied.
Our results suggest a strong correlation between perceptual similarity andpractical usefulness: the more perceptually similar discriminative features of both classes are, the lessuseful the methods become.
More importantly, the results across datasets show that on ImageNet,where attribution methods do not help, every method has a high similarity score.
This result suggeststhat after a certain threshold of perceptual similarity, attribution methods might no longer be useful,no matter how faithful or low in complexity the explanation is.
Overall, the results suggest thatthe perceptual similarity of discriminative features could explain why attribution methods fail on ImageNet.
Discussion
In summary, we conducted a large-scale human psychophysics experiment to test the utility ofexplainability methods in real-world scenarios.
Our work shows that in two of the three testedscenarios (bias detection and identification of new strategies), explainability has indeed progressedand provided meaningful assistance to human end-users.
Nevertheless, we identified a scenario(understanding failure case) for which none of the tested attribution methods were useful.
This resultis consistent with previous work [57] and highlights a fundamental challenge for XAI.
Further analysis of associated fidelity performance metrics used to guide the development of explainability methods revealed that they did not correlate with our empirical measure of utility – suggestingthat they fail to predict such failure cases.
We also investigated the possibility that the complexityof individual explanations may play a role in explaining human failures to learn to leverage thoseexplanations to understand the model and, while we found a weak correlation between complexityand our empirical measure of utility, this correlation appears too low to explain these failure cases.
Finally, we considered the possibility that fully understanding the inner-workings of a vision systemrequires understanding, not just “where” to look (as revealed by attribution maps) but also “what”features in those locations drive decisions, something not currently revealed by these methods.
Our assumption is that the need for finer “what” information should arise when diagnostic imagelocations across classes look perceptually very similar and potentially semantically related for certainclassification problems (e.g., looking at the ears or the snout to discriminate between breeds of cats anddogs) and one needs to identify what aspects of these features are driving decisions.
We computed aperceptual score for classification problems by estimating the perceptual similarity between diagnosticimage regions (as predicted by attribution methods) and found that, indeed, when this score predicts acertain level of perceptual similarity between two distinct classes, attribution methods fail to contributeuseful information to human end-users, regardless of the fidelity or complexity of the explanations.
This fundamental result highlights an intrinsic limitation of attribution methods and suggests the needfor a change of focus in the field – from quantitative improvements of current attribution methods tothe development of novel explainability methods that communicate qualitatively different knowledgeabout the inner-workings of the system.
article
machine-bias-risk-assessments-
-criminal-sentencing
[-[Yes] in sec 
-]