Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words? (2020)

Abstract: Motivated by human attention, computational attention mechanisms have been designed to help neural networks adjust their focus on specific parts of the input data. While attention mechanisms are claimed to achieve interpretability, little is known about the actual relationships between machine and human attention. In this work, we conduct the first quantitative assessment of human versus computational attention mechanisms for the text classification task. To achieve this, we design and conduct a large-scale crowd-sourcing study to collect human attention maps that encode the parts of a text that humans focus on when conducting text classification. Based on this new resource of human attention dataset for text classification, YELP-HAT, collected on the publicly available YELP dataset, we perform a quantitative comparative analysis of machine attention maps created by deep learning models and human attention maps. Our analysis offers insights into the relationships between human versus machine attention maps along three dimensions: overlap in word selections, distribution over lexical categories, and context-dependency of sentiment polarity. Our findings open promising future research opportunities ranging from supervised attention to the design of human-centric attention-based explanations.

## introduction attention-based models have become the architectures of choice for a vast number of nlp tasks including, but not limited to, language modeling @xcite , machine translation @xcite , document classification @xcite , and question answering @xcite . while attention mechanisms have been said to add interpretability since their introduction @xcite , the investigation of whether this claim is correct has only just recently become a topic of high-interest @xcite @xcite . if attention mechanisms indeed offer a more in-depth understanding of a model's inner-workings, application areas from model debugging to architecture selection would benefit greatly from profound insights into the internals of attention-based neural models. recently, @xcite , @xcite , and @xcite proposed three distinct approaches for evaluating the explainability of attention. @xcite base their work on the premise that explainable attention scores should be unique for a given prediction as well as consistent with other featureimportance measures. this prompts their conclusion that attention is not explanation. based on similar experiments on alternative attention scores, @xcite conclude that attention does not necessarily correspond to the importance of inputs. in contrast, @xcite find that attention learns a meaningful relationship between input tokens and model predictions, which cannot be easily hacked adversarially. while these works ask valuable questions, they embrace model-driven approaches for manipulating the attention weights and thereafter evaluate the post-hoc explainability of the generated machine attention. in other words, they overlook the human factor in the evaluation process -which should be integral in assessing the plausibility of the generated explanations @xcite . in this work, we adopt a novel approach to attention explainability from a human-centered perspective and, in particular, investigate to what degree machine attention mimics human behavior. more precisely, we are interested in the following research question: do neural networks with attention mechanisms attend to the same parts of the text as humans? to this end, we first collect a large dataset of human-attention maps and then compare the validated human attention with a variety of machine attention mechanisms for text classification. figure 1 displays examples of human and machine-generated attention for classifying a restaurant review's overall rating. our goal is to quantify the similarity between human attention and machine-generated attention scores. measuring this similarity is non-trivial and is not appropriately captured by an existing similarity metric (e.g., euclidean) between two vectors for the following reasons. a binary human attention vector does not solely denote which tokens are given higher importance but also implies information about the underlying grammatical structure and linguistic construction. for example, whether or not adjectives tend to be high-importance is encoded in the attention weights as well. further, it is well known that human attention is itself subjective: given the same text and task, human annotators may not always agree on which words are important. that is, one single human's attention should rarely be regarded as the ground-truth for attention. given this objective, we use crowd-sourcing to collect a large set of human attention maps. we provide a detailed account of the iterative design process for our data collection study in §3. we design new metrics that quantify the similarity between machine and human attention from three perspectives ( §4): behavioral similarity measures the number of common words selected by human and machine discerning if neural networks with attention mechanisms attend to the same parts of the text as humans. humans associate certain lexical categories (e.g., adjectives) with a sentiment more heavily. lexical (grammatical) similarity identifies if machine attention favors similar lexical categories with humans. a high lexical similarity shows that the attention mechanism learns similar language patterns with humans. context-dependency quanitifies sentiment polarity of word selections. we then employ these metrics to compare attention maps from a variety of attention-based recurrent neural networks (rnn). we find that bi-directional rnns with additive attention demonstrate strong similarities to human attention for all three metrics. in contrast, uni-directional rnns with attention differ from human attention significantly. finally, as the text length increases, and with it, the prediction task becomes more difficult, both the accuracy of the models and similarity between human and machine decrease. our contributions are as follows: our collected yelp-hat (human attention) dataset is publicly available as a valuable resource to the nlp community. ## preliminaries on attention maps in this section, we define the concepts of human attention map and machine attention map. definition 2.1. attention map. an attention map (am) is a vector where each entry in sequence is associated with a word in the corresponding position of the associated text. the value of the entry indicates the level of attention the corresponding word receives with respect to a classification task. definition 2.2. human attention map. a human attention map (ham) is a binary attention map produced by a human, where each entry with a set-bit indicates that the corresponding word receives high attention. definition 2.3. machine attention map. a machine attention map (mam) is an attention map generated by a neural network model. if computed through soft-attention, a mam corresponds to an am of continuous values, that capture a probability distribution over the words. if computed through hard-attention, a mam is a binary am. we now introduce the application of aggregation operators to coalesce hams by multiple annotators into aggregated hams. definition 2.4. consensus attention map. if multiple hams exist for the same text, a consensus attention map (cam) is computed through a bitwise and operation of the hams. definition 2.5. super attention map. if multiple hams exist for the same text, a super attention map (sam) is computed by a bitwise or operation of the hams. ## attention map similarity framework we quantify the similarity between hams and mams through our similarity framework that contains three new metrics as described in this section. ## is machine attention similar to human attention? ## related work a large body of work has been using attention mechanisms to attempt to bring 'interpretability' to model predictions @xcite @xcite . however, they only assess the produced attention maps qualitatively by visualizing a few hand-selected instances. recently, researchers began to question the interpretability of attention. @xcite and @xcite argue that if alternative attention distributions exist that produce similar results to those obtained by the original model, then the original model's attention scores cannot be reliably used to explain the model's prediction. they empirically show that achieving such alternative distributions is possible. in contrast, @xcite find that attention learns a meaningful relationship between input tokens and model predictions which cannot be easily hacked adversarially. @xcite conducted the first quantitative assessment of computational attention mechanisms for the visual question answering (vqa) task. similar to our work, they collect a human attention dataset, then measure the similarity of human and machine attention within the context of vqa. this vqa-hat dataset now provides a fertile research vehicle for researchers in computer vision for studying the supervision of the attention mechanism @xcite . the development of a similar dataset and an in-depth quantitative evaluation for text to advance nlp research is sorely lacking. in a concurrent and independent work, @xcite collects the eraser dataset for human annotations of rationales. while eraser includes multiple datasets for a number of nlp tasks with relatively small amounts of data for each, we focus on text classification and collect a large amount of data on a different corpus. ## discussion recent papers, including our work, take strides at answering the question if attention is interpretable. this is complicated by the fact that "interpretability" remains a not well-defined concept. attention adds transparency. lipton (2018) defines transparency as overall human-understanding of a model, i.e., why a model makes its decisions. under this definition, attention scores can be seen as partial transparency. that is, they provide a look into the inner workings of a model, in that they produce an easily-understandable weighting of hidden states @xcite . attention is not faithful. whether adversarial attention scores exist that result in the same predictions as the original attention scores helps us understand if attention is faithful. with their empirical analyses, @xcite and @xcite show that attention is not faithful. rationale models for human-like explanations. @xcite argues that explanations are post-hoc descriptions of how a system came to a given conclusion. this raises the question of what makes a good explanation of the behavior of a machine learning system. one line of research offers these explanations in the form of binary rationales, namely, explanations that plausibly justify a model's actions @xcite . our approach at attention as human-like explanations. in claiming attention is explanation, it is seen to mimic humans in rationalizing past actions. in our work, we approach interpretability from this human-centric perspective. we develop a systematic approach to either support or refute the hypothesis that attention corresponds to humanlike explanations for model behavior. based on our comparative analyses, we provide initial answers to this important question by finding insights into the similarities and dissimilarities of attention-based architectures to human attention. towards additional tasks beyond text classification. confidently concluding whether attention mimics human requires tremendous efforts from many researchers with human data to be collected via a well-designed data collection methodology, both labor-intensive and costly task. in this work, we thus focus on one task, namely, sentiment classification, and collect ham for this task and on a single dataset. we invite other researchers to continue this line of research by exploring other tasks (e.g., question answering). next steps in attention research. our work opens promising future research opportunities. one is to supervise attention models explicitly. attention mechanisms themselves are typically learned in an unsupervised manner. however, initial research offers compelling evidence for the success of supervised attention models @xcite in the computer vision area. also, attention has the potential to be leveraged for both making predictions and concurrently producing human-centric explanations similar to rationalebased architectures. ## conclusion to gain a deeper understanding of the relationships between human and attention-based neural network models, we conduct a large crowd-sourcing study to collect human attention maps for text classification. this human attention dataset represents a valuable community resource that we then leverage for quantifying similarities between human and attention-based neural network models using novel attention-map similarity metrics. our research not only results in insights into significant similarities between bidirectional rnns and human attention, but also opens the avenue for promising future research directions. another example is provided in figure 3 , demonstrating the attention maps provided by two different annotators for the same review. this is an extreme example of the subjectivity of human attention. the first annotator only highlights individual words with the strongest cues of sentiment, whereas the second annotator sometimes selects entire sentences when they indicate a sentiment.