Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methods (2023)

Abstract: Saliency maps can explain a neural model’s predictions by identifying important input features. They are difficult to interpret for laypeople, especially for instances with many features. In order to make them more accessible, we formalize the underexplored task of translating saliency maps into natural language and compare methods that address two key challenges of this approach – what and how to verbalize. In both automatic and human evaluation setups, using token-level attributions from text classification tasks, we compare two novel methods (search-based and instruction-based verbalizations) against conventional feature importance representations (heatmap visualizations and extractive rationales), measuring simulatability, faithfulness, helpfulness and ease of understanding. Instructing GPT-3.5 to generate saliency map verbalizations yields plausible explanations which include associations, abstractive summarization and commonsense reasoning, achieving by far the highest human ratings, but they are not faithfully capturing numeric information and are inconsistent in their interpretation of the task. In comparison, our search-based, model-free verbalization approach efficiently completes templated verbalizations, is faithful by design, but falls short in helpfulness and simulatability. Our results suggest that saliency map verbalization makes feature attribution explanations more comprehensible and less cognitively challenging to humans than conventional representations.

## introduction feature attribution methods, or (input) saliency methods, such as attention-or gradient-based attribution, are the most prominent class of methods for generating explanations of nlp model behavior @xcite and can be used to produce word-level importance scores 1 code and data at https://github.com/dfki-nlp/smv . natural language explanations (nles), on the other hand, exceed other explainability methods in plausibility @xcite @xcite , accessibility @xcite , and flexibility @xcite , i.e. they can be adapted to both different target tasks and different audiences. most previous approaches in generating nles depend on datasets of humanannotated text highlights @xcite @xcite or carefully constructed gold rationales for supervised training @xcite , which are costly to obtain and taskspecific. alignment of model rationales with very few human-acceptable gold rationales may raise issues of trust @xcite and the models trained on them may suffer from hallucinations @xcite . in this work, we revisit and formalize the task of verbalizing saliency maps, i.e. translating the output of feature attribution methods into natural language @xcite @xcite . verbalizations can describe relations between words and phrases and their associated saliency scores. contrary to conventional heatmap visualizations, we can adjust the comprehensiveness of an explanation more precisely and infuse it with additional semantics such as word meanings, concepts, and context about the task. we find that verbalization also comes with a few caveats: similar to human explainers, who communicate only the most relevant explanations to avoid cognitive overload of the recipient (hilton, 2017 @xcite , verbalization methods need to address the problem of deciding "what" to say, i.e. selecting the most informative and useful aspects of the saliency maps and communicating them in a concise manner. we therefore compare different methods for verbalizing saliency maps: supervised rationales, prompting llms, and model/trainingfree templates. we address the problem of saliency map verbalization (smv) with the following contributions: • we formalize the underexplored task of smv and establish desiderata, i.e. simulatability, explainerfaithfulness, plausibility, and conciseness ( §2.1); • we conduct a comparative study on various representations of feature attribution in two text classification setups, measuring the effects of verbalizations methods on both automated (explainerfaithfulness) and human evaluation metrics (simulatability, helpfulness, ease of understanding) ( §3, §5). ## methods to complement heatmap visualizations and extractive rationales, we propose and analyze two additional verbalization methods: model-free ( §4.1, fig. 2 ) and instruction-based ( §4.2, fig. 3 ) saliency map verbalization. ## data we choose datasets that cover a selection of english-language text classification tasks. in particular, we select imdb @xcite for sentiment analysis, and ag news @xcite for topic classification. we retrieve predictions from bert models on the test partitions of imdb and ag news made available through textattack @xcite and their integrated gradients @xcite explanations with 25 samples exactly as they appear in thermostat @xcite . we then take subsets (imdb: @xmath0, ag news: @xmath1. d) that make the tasks more manageable for annotators. each annotator was shown 340 explanations consisting of equal amounts of each type of representation or rationale. we randomize the order in which they are presented to the annotators. every instance was evaluated by seven different annotators. ## results human evaluation tab. 2 shows that both kinds of smvs are generally easier to understand (b2) than heatmaps or extractive rationales. in a postannotation questionnaire, we asked 8 out of 10 participants 14 questions about both types of smvs. tab. 1 lists the results. while template-based explanations are preferred in being less time-consuming, we can see that gpt-generated verbalizations outperform them in all other aspects. unsurprisingly, associations and summarizations are the preferred characteristics of verbalizations. @xcite , a feature attribution explanation aggrecoverage of the verbalization fig. 4 and app. a show that smv gpt focuses less on the actual most important tokens that might not be intuitive for recipients, such as function words. the subset of instances with higher-than-average coverage according to smv templ (cov(s vt ) ↗ ) is substantially easier to simulate (imdb) and elicits the highest ratings and accuracies from annotators. we utilize this as a proxy for (low) complexity of s, because usually only a single or few tokens that are very salient make these explanations easy to decipher in most representations. therefore, we conducted an automated simulatability evaluation on all smv types, documented in appendix b, confirming the suspicions about the faithfulness of gpt verbalizations. ## related work to our knowledge, the only previous saliency map verbalization approach is by @xcite who used lime explanations and a template-based nlg pipeline on a credit dataset. while they mostly included numerical values in explanations, we focus on most important features and free-text rationales, because humans are more interested in reasoning than in numerical values @xcite . @xcite created a dataset of tables summarizing the performance metrics of a text classifier and trained a neural module to automatically generate accompanying texts. the hci community highlighted the advantages of verbalization as a complementary medium to visual explanations @xcite @xcite . @xcite advocated for adding concepts and associations to make explanations more understandable, particularly in contrastive setups. ## conclusion we conducted a comparative study on explanation representations. we formalized the task of translating feature attributions into natural language and proposed two kinds of saliency map verbalization methods. instruction-based verbalizations outperformed all other saliency map representations on human ratings, indicating their summarization and contextualization capabilities are a necessary component in making saliency maps more accessible to humans, but they are still unreliable in terms of ensuring faithfulness and are dependant on a closedsource black-box model. we find that templatebased saliency map verbalizations reduce the cognitive load for humans and are a viable option to improve on the ease of understanding of heatmaps without the need for additional resources.