Logic Traps in Evaluating Attribution Scores (2022)

Abstract: Modern deep learning models are notoriously opaque, which has motivated the development of methods for interpreting how deep models predict. This goal is usually approached with attribution method, which assesses the influence of features on model predictions. As an explanation method, the evaluation criteria of attribution methods is how accurately it reflects the actual reasoning process of the model (faithfulness). Meanwhile, since the reasoning process of deep models is inaccessible, researchers design various evaluation methods to demonstrate their arguments. However, some crucial logic traps in these evaluation methods are ignored in most works, causing inaccurate evaluation and unfair comparison. This paper systematically reviews existing methods for evaluating attribution scores and summarizes the logic traps in these methods. We further conduct experiments to demonstrate the existence of each logic trap. Through both theoretical and experimental analysis, we hope to increase attention on the inaccurate evaluation of attribution scores. Moreover, with this paper, we suggest stopping focusing on improving performance under unreliable evaluation systems and starting efforts on reducing the impact of proposed logic traps.

## introduction the opaqueness of deep models has grown in tandem with their power @xcite , which has motivated efforts to interpret how these black-box models work @xcite . post-hoc explanation aims to explain a trained model and reveal how the model arrives at a decision @xcite . this goal is usually approached with attribution method, which assesses the influence of features on model predictions as shown in figure 1 . recent years have witnessed an increasing number of attribution meth-ods being developed. for example, erasure-based method calculate attribution scores by measuring the change of output after removing target features @xcite @xcite ; gradient-based method uses gradients to study the influence of features on model predictions @xcite @xcite ; meanwhile, these methods also received much scrutiny, arguing that the generated attribution scores are fragile or unreliable @xcite @xcite . as an explanation method, the evaluation criteria of attribution methods should be how accurately it reflects the true reasoning process of the model (faithfulness), not how convincing it is to humans (plausibility) @xcite . meanwhile, since the reasoning process of deep models is inaccessible, researchers design various evaluation methods to support their arguments, some of which appear valid and are widely used in the research field. for example, meaningful perturbation is used for making comparison in many works @xcite @xcite @xcite . the philosophy of meaningful perturbation is simple, i.e., modifications to the input instances, which are in accordance with the generated attribution scores, can bring about significant differences to model predictions if the attribution scores are faithful to the target system. however, some crucial logic traps existing in these evaluation methods are ignored in most works, causing inaccurate evaluation and unfair comparison. for example, we found that we can manipulate the evaluation results when using meaningful perturbation to make comparisons: by choosing the modification strategy, we can assign any of the three candidate attribution methods as the best method. the neglect of these traps has damaged the community in many aspects: first, the existence of logic traps will lead to an inaccurate evaluation and unfair comparison, making the conclusions unreliable; second, the wide use of evaluation metrics with logic traps brings pressure to newly proposed works, requiring them to compare with other works using the same metrics; last, the over-belief in existing evaluation metrics encourages efforts to propose more accurate attribution methods, notwithstanding the evaluation system is unreliable. in this paper, we systematically review existing methods for evaluating attribution scores and categorize them into classes. we summarize the logic traps in these methods and further conduct experiments to demonstrate the existence of each logical trap. though strictly accurate evaluation metrics for attribution methods might be a "unicorn" which will likely never be found, we should not just ignore logic traps in existing evaluation methods and draw conclusions recklessly. through both theoretical and experimental analysis, we hope to increase attention on the inaccurate evaluation of attribution scores. moreover, with this paper, we suggest stopping focusing on improving performance under unreliable evaluation systems and starting efforts on reducing the impact of proposed logic traps. ## evaluation methods and corresponding logic traps 1 : heat maps for a portion of a yelp review generated by different attribution techniques. the example and results are taken from @xcite . furthermore, resorting to human-annotated explanations, works can also evaluate attribution methods quantitatively in evaluation 1. for example, the sst-2 @xcite corpus provides not only sentence-level labels, but also five-class word-level sentiment tags ranging from very negative to very positive. thus, many works @xcite @xcite perform quantitative evaluation of attribution scores by comparing them with the word-level tags in sst-2. logic trap 1: the decision-making process of neural networks is not equal to the decisionmaking process of humans. first, we cannot completely deny the rationality of evaluation 1. since many attribution methods work without any human-annotated information, such as erasure-based and gradient-based methods, the similarity between human-annotated explanations and generated attribution scores can be seen as drawing from the reasoning process of target models. however, since the deep model often rely on unreasonable correlations, even when producing correct predictions, attribution scores preposterous to humans may reflect the reasoning process of the deep model faithfully. thus we cannot deny the validity of an attribution score through its inconsistency to human-annotated explanations and cannot use human-annotated explanations to conduct a quantitative evaluation.