Neighboring Words Affect Human Interpretation of Saliency Explanations (2023)

Abstract: Word-level saliency explanations (“heat maps over words”) are often used to communicate feature-attribution in text-based models. Recent studies found that superficial factors such as word length can distort human interpretation of the communicated saliency scores. We conduct a user study to investigate how the marking of a word’s *neighboring words* affect the explainee’s perception of the word’s importance in the context of a saliency explanation. We find that neighboring words have significant effects on the word’s importance rating. Concretely, we identify that the influence changes based on neighboring direction (left vs. right) and a-priori linguistic and computational measures of phrases and collocations (vs. unrelated neighboring words).Our results question whether text-based saliency explanations should be continued to be communicated at word level, and inform future research on alternative saliency explanation methods.

## introduction in the context of explainability methods that assign importance scores to individual words, we are interested in characterizing the effect of phrase-level features on the perceived importance of a particular word: text is naturally constructed and comprehended in various levels of granularity that go beyond the word-level @xcite . for example (figure 1 ), the role of the word "york" is contextualized by the phrase "new york" that contains it. given an explanation that attributes importance to "new" and "york" separately, what is the effect of the importance score of "new" on the explainee's understanding of the importance "york"? our study investigates this question. current feature-attribution explanations in nlp mostly operate at word-level or subword-level @xcite ; ribeiro the company has been headquartered in new york since its ipo in the year 2013 . figure 1 : illustration of the user study. we ask laypeople to rate the perceived importance of words following a word-importance explanation (grey). then we analyze the effect of the importance of neighboring words on this interpretation, conditioned on the relationship between the words across various measures (orange). @xcite . previous work investigated the effect of word and sentence-level features on subjective interpretations of saliency explanations on text @xcite -finding that features such as word length and frequency bias users' perception of explanations (e.g., users may assign higher importance to longer words). it is not trivial for an explanation of an ai system to successfully communicate the intended information to the explainee @xcite @xcite . in the case of feature-attribution explanations @xcite , which commonly appear in nlp as explanations based on word importance @xcite , we must understand how the explainee interprets the role of the attributed inputs on model outputs @xcite . research shows that it is often an error to assume that explainees will interpret explanations "as in-tended" @xcite . the study involves two phases (figure 1 ). first, we collect subjective self-reported ratings of importance by laypeople, in a setting of color-coded word importance explanations of a fact-checking nlp model (section 2, figure 2 ). then, we fit a statistical model to map the importance of neighboring words to the word's rating, conditioned on various a-priori measures of bigram constructs, such as the words' syntactic relation or the degree to which they collocate in a corpus @xcite . we observe significant effects (section 4) for: 1. left-adjacency vs. right-adjacency; 2. the difference in importance between the two words; 3. the phrase relationship between the words (common phrase vs. no relation). we then deduce likely causes for these effects from relevant literature (section 5). we are also able to reproduce results by @xcite in a different english language domain (section 3). we release the collected data and analysis code. 1 we conclude that laypeople interpretation of word importance explanations in english can be biased via neighboring words' importance, likely moderated by reading direction and phrase units of language. future work on feature-attribution should investigate more effective methods of communicating information @xcite , and implementations of such explanations should take care not to assume that human users interpret word-level importance objectively. ## study specification our analysis has two phases: collecting subjective interpretations of word-importances from laypeople, and testing for significant influence in various properties on the collected ratings-in particular, properties of adjacent words to the rated word. ## reproducing prior results our study is similar to the experiments of @xcite who investigate the effects of wordlevel and sentence-level features on importance perception. thus, it is well-positioned to attempt a reproduction of prior observations, to confirm whether they persist in a different language domain: medium-form wikipedia texts vs. shortform restaurant reviews in @xcite . the result is positive: we reproduce the previously reported significant effects of word length, display index (i.e., the position of the rated instance within the 150 sentences), capitalization, and dependency relation for randomized explanations as well as shap-value explanations (details in appendix a). this result reinforces prior observations that human users are at significant risk of biased perception of saliency explanations despite an objective visualization interface. ## neighbor effects analysis in the following, we present our results for our two experiments using (a) random saliency values and (b) shap values. ## theoretical grounds in psychology the assimilation effect is, of course, intuitive-it simply mean that neighbors' importance "leaks" from neighbor to the rated word for strong bigram relationships. but is there precedence for the observed assimilation and contrast effects in the literature? how do they relate to each other? psychology investigates how a prime (e.g., being exposed to a specific word) influences human judgement, as part of two categories: assimilation (the rating is "pulled" towards the prime) and contrast (the rating is "pushed" away from the prime) effects (i.a., @xcite . @xcite demonstrate how global processing (e.g. looking at the overall structure) vs. local processing (e.g., looking at the details of a structure) leads to assimilation vs. contrast. we argue that some of our observations can be explained with their model: multi-word phrase neighbors may induce global processing that leads to assimilation (for example, in the randomized explanation experiments, left neighbors) while other neighbors (in the randomized explanation experiments, right neighbors and unrelated left neighbors) induce local processing that leads to contrast. future work may investigate the properties that induce global processing in specific contexts. ## conclusions we conduct a user study in a setting of laypeople observing common word-importance explanations, as color-coded importance, in the english wikipedia domain. in this setting, we find that when the explainee understands the attributed importance of a word, the importance of other words can influence their understanding in unintended ways. common wisdom posits that when communicating the importance of a component in a featureattribution explanation, the explainee will understand this importance as it is shown. we find that this is not the case: the explainee's contextualized understanding of the input portion-for us, a word as a part of a phrase-may influence their understanding of the explanation.