Evaluating Saliency Explanations in NLP by Crowdsourcing (2024)

Abstract: Deep learning models have performed well on many NLP tasks. However, their internal mechanisms are typically difficult for humans to understand. The development of methods to explain models has become a key issue in the reliability of deep learning models in many important applications. Various saliency explanation methods, which give each feature of input a score proportional to the contribution of output, have been proposed to determine the part of the input which a model values most. Despite a considerable body of work on the evaluation of saliency methods, whether the results of various evaluation metrics agree with human cognition remains an open question. In this study, we propose a new human-based method to evaluate saliency methods in NLP by crowdsourcing. We recruited 800 crowd workers and empirically evaluated seven saliency methods on two datasets with the proposed method. We analyzed the performance of saliency methods, compared our results with existing automated evaluation methods, and identified notable differences between NLP and computer vision (CV) fields when using saliency methods. The instance-level data of our crowdsourced experiments and the code to reproduce the explanations are available at https://github.com/xtlu/lreccoling_evaluation.

## introduction recently, the development of deep neural network (dnn) models for natural language processing, such as bert @xcite , and gpt-3 @xcite , has attracted considerable attention as a topic of active research. given that dnns can recognize important patterns in large and intricate datasets, dnn models exhibited significant achievements on various tasks for different applications. however, most deep models are black boxes, i.e., the relationship between their inputs and outputs is difficult to understand. in most low-risk tasks, errors made by dnns do not have severe impacts; for example, in a recommendation system designed to suggest fiction novels, the impact of recommendation errors is relatively low. however, in some situations, the dnn model performing an nlp task could cause psychological harm to users. for example, an online chat service could output unethical or offensive sentences. to avoid this, one of the most fundamental and effective solution is to understand the decisionmaking process of the relevant models. therefore, as the complexity of dnn models increases, the need for research on the interpretation of the output of such models becomes increasingly pressing. various interpretation methods have emerged to attempt to understand dnns @xcite @xcite , particularly for black-box predictions made by alreadytrained neural networks. one of the major approaches to this problem involves the estimation of the influence of a subset of input features on the predictions of a model. by understanding the important features, the model can be improved, its predictions can be trusted more, and undesirable behaviors can be isolated @xcite . saliency explanations that output a contribution score for each feature in an input have been widely adopted in dnns. important features in input can be found by ranking contributions scores. thus, saliency methods can be regarded as a general solution for the issue of black-box models, and various saliency methods have been proposed such as vanilla gradients @xcite @xcite , integrated gra-dients @xcite , and local interpretable model-agnostic explanations (lime) @xcite . the explanations rendered by various saliency methods often diverge; for example, we apply saliency methods on 100 movie review samples and found that the overlap rates, i.e. consistency of different saliency methods, were relatively low, as shown in along these lines, many automated evaluation metrics have been proposed to verify different interpretable models @xcite @xcite . however, such automated evaluation measures are not guaranteed to correctly evaluate the interpretability of saliency methods. there are essential differences between machine cognition and human cognition. for example, @xcite found that changing a single pixel in an image can change the output of models, whereas such changes are generally completely imperceptible to humans. therefore, the interpretability of saliency methods should ultimately be decided by humans and a high rating by an automated evaluation metric does not necessarily mean a high interpretability for humans. @xcite found that neighboring words could affect saliency explanations for human. de langis and kang (2023) leveraged human eye-tracking to study saliency explanations. @xcite discussed human-machine agreements on different models for a gradient-based saliency method. however, evaluating saliency methods from the perspective of human intelligence has not been fully investigated. crowdsourcing provides an efficient and economical way to solve various tasks by utilizing human intelligence. for example, @xcite proposed a mechanism of trust evaluation on edge computing using crowdsourcing. @xcite exploited a crowdsourcing platform to evaluate human visual attention paths predicted by models. intuitively, we consider that a better saliency method should assign higher scores to more important words, enabling humans to correctly infer properties (e.g., categories) of the original text by the given several top important words. the key idea of this study is to apply crowdsourcing to evaluating saliency methods by whether crowd workers could distinguish labels of text based on the given top important words, as inspired by the measurements proposed by @xcite on image classification tasks. based on the experimental results of the proposed evaluation method, we reach a list of findings on the saliency explanation methods in the nlp area. the contributions of this study are summarized as follows. 1) we propose a novel human-based method to evaluate saliency methods in nlp tasks by crowdsourcing. we recruited 800 workers from a crowdsourcing platform to evaluate seven saliency methods on two datasets and verified the reliability of the data of annotations provided by crowd workers. 2) we found that integrated gradient, a saliency method, performed best in our experiments and some saliency methods were not significantly better than random baseline. 3) we compared our results based on human evaluation with existing automated evaluation methods and found that there were differences between ours and existing methods. 4) we found a phenomenon called flip, which makes saliency methods more unreliable in nlp than cv tasks. 5) we evaluated saliency methods on misclassified samples and found that saliency methods were also effective for misclassified samples. ## crowd-based evaluation method we evaluated saliency methods by showing the top important words given by the method to crowd workers. in the web interface, crowd workers are asked to perform a text classification task, i.e., assigning a label to a text from a set of labels based on the words provided. figure 1 shows an example of a task. we denote by l @xmath0, 0, 1, 2, ..., c} the annotation given by crowd worker m in the task t ijk of showing top-k important words by the j-th saliency method for sample i, where @xmath1, 2, ..., n }, n is the number of samples, @xmath2, k is the number of different top-k words settings, and c is the number of classes. l @xmath3, l @xmath4, 2, ..., c} indicates that crowd worker m selected label c in the task t ijk . we denote by a @xmath5, 1} the results of whether task t ijk can be correctly recognized by crowd workers, where a @xmath6. we aggregate annotations from multiple workers by majority voting because of the large amount of tasks and the very sparse annotations with 3900 4000 = 97.5% missing rate. for a sample i with ground truth label c, the aggregation can be represented by the following equation. next, we measure the performance of saliency methods in terms of the aggregation results a ijk . we consider that if a saliency explanation is sufficiently "interpretable", the accuracy, i.e., probability of labels can be correctly recognized more accurately by humans based on the given top-k important words. the accuracy of the j-th saliency method for the top-k words. the performance of the j-th saliency method s j can be considered as the weighted average of p jk of each k. when k is relatively small, obtaining higher accuracy is more difficult than when k is large, so we assign smaller k larger weights. we use the mean accuracy of all saliency methods of the top-k words as the difficulty of the top-k words. the performance score can be represented as the following equation. the evaluation procedure consists of the following steps: ## experiments we conducted experiments to answer the following three questions. 1) which saliency methods performed best in human evaluations? 2) do our results agree with the previous studies? 3) what is different between texts and images in the saliency methods? 4) what is the findings of evaluating misclassified cases? ## conclusion in this study, we proposed an approach to evaluate saliency explanations in natural language process the scale of our experiments was limited by the cost of crowdsourcing because real crowdsourcing experiments are costly and adding more datasets or models will dramatically increase the cost. although the number of datasets is relatively small and the results were only evaluated on the bertbase model, the main purpose of this work is to claim that there are differences between human evaluation and automated evaluations and there are indeed differences in our experiments of one model and two datasets. as long as there are differences in one model, there are differences. although more experiments will support our claim, we would like to caution further studies when using automated evaluation methods. four takeaways can be summarized as follows based on the results. 1) integrated gradient performed best in our experiments. all attention, last atten-tion and vanilla gradient performed significantly better than the random baseline on both two datasets. deeplift and lime performed close to or worse than the baseline, probably because the contribution scores were calculated on token embeddings but not directly on tokens (words), which leads to additional complexity in the calculation.