Understanding Attention for Text Classification (2020)

Abstract: Attention has been proven successful in many natural language processing (NLP) tasks. Recently, many researchers started to investigate the interpretability of attention on NLP tasks. Many existing approaches focused on examining whether the local attention weights could reflect the importance of input representations. In this work, we present a study on understanding the internal mechanism of attention by looking into the gradient update process, checking its behavior when approaching a local minimum during training. We propose to analyze for each word token the following two quantities: its polarity score and its attention score, where the latter is a global assessment on the token’s significance. We discuss conditions under which the attention mechanism may become more (or less) interpretable, and show how the interplay between the two quantities can contribute towards model performance.

## introduction attention mechanism @xcite has been used as an important component across a wide range of nlp models. typically, an attention layer produces a distribution over input representations to be attended to. such a distribution is then used for constructing a weighted combination of the inputs, which will then be employed by certain downstream modules. recently, several research efforts on investigating the interpretability of attention on tasks such as text classification, question answering, and natural language inference @xcite @xcite have been conducted. one of their important arguments was whether the attention distribution could adequately reflect the significance of inputs. to answer this question, they designed a series of metrics and conducted corresponding experiments. in their approaches, they were mainly observing how the attention may impact the outputs on the pre-trained models by changing some elements in the inputs. while such approaches have resulted in interesting findings, the attention mechanism itself remains a black box to us -it is still largely unclear what are the underlying factors that may have an impact on the attention mechanism. when analyzing the results of a typical model with attention on the text classification tasks, we noticed that in some instances, many of the word tokens with large attention weights were adjectives or adverbs which conveyed explicit signals on the underlying class label. on the other hand, in some other instances, we also noticed that such useful words may not always be able to receive significant attention weights, especially under certain configurations of hyperparameters, making the attention mechanism less interpretable. such observations lead to several important questions. first, the attention weight for a word token appears to be the relative measurement to its significance, and is largely local and instance specific. would there be an instance-independent quantity to assess the corpus-level importance of a word token? and if so, what role would such a quantity play in terms of interpreting the overall attention mechanism? second, when the attention mechanism appears to be less interpretable, how would the underlying model be affected in terms of performance? in this work, we focus on answering the above questions. we argue that the attention scores (rather than attention weights) are able to capture the global, absolute importance of word tokens within a corpus. we present a study to figure out the underlying factors that may influence such attention scores under a simple neural classification model. inspired by @xcite , we analyzed the gradients as well as the updates of intermediate variables in the process of gradient descent, and found that there exist some implicit trends on the intermediate variables related to attention: the degree of association between a word token and the class label may impact their attention scores. we argue that when certain hyperparameters are properly set, tokens with strong polarity -high degree of association with specific labels, would likely end up with large attention scores, making them more likely to receive large attention weights in a particular sentence. while in such scenarios, the attention mechanism would appear to be more interpretable, we also discuss scenarios where the attention weights may become less interpretable, and show how the polarity scores, another important token-level quantity, will play their roles in the overall model in terms of contributing towards the model performance. ## related work research on interpretability of neural models has received significant attention recently. one approach was using visualization to explore patterns that exist in the intermediate representations of neural networks. @xcite visualized the image-specific class saliency on image classification tasks using learnt convnets, and displayed the features captured by the neural networks. @xcite proposed visualization methods to look into the neural representations of the embeddings from the local composition, concessive sentences, clause composition, as well as the saliency of phrases and sentences, and illustrated patterns based on the visualizations. an erasure method was also adopted to validate the importance of different dimensions and words. @xcite analyzed the attention structure on the transformer @xcite language model as well as gpt-2 @xcite pre-trained model. another approach to understanding neural approaches is to conduct theoretical analysis to investigate the underlying explanations of neural models. one example is the work of @xcite , which regarded the word embedding learning task as an optimization problem, and found that the training process of the skip-gram model @xcite ) can be explained as implicit factorization of a shifted positive pmi (pointwise mutual information) matrix. recently, several research efforts have focused on the interpretability of the attention mechanism. @xcite raised the question on the explainability of feature importance as captured by the attention mechanism. they found the attention weights may not always be consistent with output figure 1 : classification architecture with attention the feature importance from the human perspective in tasks such as text classification and question answering. @xcite also carried out an analysis on the interpretability of the attention mechanism, with a focus on the text classification task. they conducted their study in a cautious way with respect to defining interpretability and the research scope. the paper concluded that the attention weights are noisy predictors of importance, but should not be regarded as justification for decisions. @xcite suggested that the notion of explanation needs to be clearly defined, and the study of the explanation requires taking all components of a model into account. their results indicated that prior work could not disprove the usefulness of attention mechanisms with respect to explainability. moreover, @xcite and @xcite examined the multi-head self-attention mechanism on transformer-based models, particularly the roles played by the heads. our work and findings are largely consistent with such findings reported in the literature. we believe there are many factors involved when understanding the attention mechanism. inspired by qian (1999), which investigated the internal mechanism of gradient descent, in this work we focus on understanding attention's internal mechanism. ## classification model with attention we consider the task of text classification, with a specific focus on binary classification. 2 the architecture of the model is depicted in figure 1 . there are various attention mechanisms introduced in the field @xcite . two commonly used mechanisms are the additive attention @xcite and scaled dot-product attention @xcite . in this work, we will largely focus our analysis on the latter approach (but we will also touch the former approach later). consider an input token sequence of length n: @xmath0, e 2 , . . . , e n , where e j is the j-th input token whose representation before the attention layer is h @xmath1. the attention score for the j-th token is: the corresponding attention weight would be: the complete input sequence is represented as: during training, assume we have a training set @xmath2, y (1) ), (x (2) , y (2) ), . . . , (x (m) , y (m) )} with m labeled instances. our overall loss is: the instance-level polarity score s can also be written as: here, we have introduced the token-level polarity score s j for the input token representation h j : from here we can observe that the instance-level polarity score of the input sequence can be interpreted as the weighted sum of the token-level polarity scores, where the weights are given by the attention weights (α j for h j ). such attention weights measure the relative importance of the token within a specific input sequence. on the other hand, the attention score a j captures the absolute importance of the token. we believe such absolute measurements to the significance of words may be playing a more crucial role (than attention weights) when understanding the attention mechanism. thus, unlike many previous research efforts, we will instead focus on the understanding of attention scores in this work. in this paper, we will mainly investigate a simple neural model where h @xmath3. here e j is the word embedding for the j-th input token. in other words, we assume the word embeddings are used as the inputs to the attention layer. detailed discussions on other assumptions on h j can be found in the supplementary material. ## analysis we conduct some analysis in this section to understand how the attention mechanism works for the task of text classification. first, let us consider the following 3 different types of tokens: the gradient descent algorithm that minimizes a loss ℓ could be interpreted as the integration of the gradient flow equation using euler's method @xcite , written as: ## experiments we conducted experiments on four text classification datasets foot_4 . the statistics of the datasets are shown in if γ @xmath4.5, 1) and f + @xmath5, the token will be regarded as a "positive token". if γ @xmath6, -0.5) 9 for multi-class classification, we determined the polarity of each token based on the relative frequency of each token with respect to each label. for each token, we calculated the frequency distribution across the labels that they appear in. if the largest element of the distribution is above a given threshold, we will regard the token as a polarity one. and f - @xmath7, the token will be regarded as a "negative token". if γ @xmath8.1, 0.1) and |f + e -f - e | < 5, the token will be regarded as a "neutral token". 10 we ran the experiments using different scaling factors λ on the models with the scaled dot-product attention (dp) and additive attention (ad) respectively. for the former, we also investigated the performances on the models with a lstm (dp-l) or an affine transformation layer (dp-a) as the input encoder. 11 the adagrad optimizer @xcite was used for gradient descent. dropout @xcite ) was adopted to prevent overfitting. all the parameters were learned from scratch to avoid the influence of prior information. for the same reason, while we may be able to use pretrained word embeddings, we chose to initialize word embeddings with a uniform distribution from -0.1 to 0.1 with a dimension @xmath9. the results are shown in figure 2 shows the results for polarity scores and attention scores for the first 3 datasets, when λ is set to a moderate value of 10 (i.e., √ d). these results are consistent with our analysis. it can be observed that generally positive tokens have positive polarity scores while negative tokens have negative polarity scores. neutral tokens typically have polarity scores around zero. it can also be observed that both the positive and negative tokens generally have larger attention scores than the neutral tokens. we also tried to set a constraint on v w by introducing a regularization term to minimize it in the learning process. we found doing so will generally encourage the attention model to produce more interpretable attention scores -for example, even when λ was large, both the positive and negative tokens ended up with positive attention scores that were generally larger than those of the neutral tokens. however, empirically we did not observe a significant improvement in test performance. see the supplementary material for details. we examined the attention scores on the 20news ii dataset which consists of 3 labels. as shown in figure 3c , polarity tokens that are strongly associated with specific labels are still likely to have larger attention scores than those of neutral tokens. to understand whether there are similar patterns for the polarity and attention scores when using the additive attention models, we replaced the scaled dot-product attention layer with the additive attention layer and ran experiments on the sst dataset. the results are shown in figure 4 , which are similar to those of our scaled dot-product attention model. furthermore, we analyzed the relationship between the global attention scores and the local attention weights. we collected all the attention weights on the test set of sst for the positive, negative and pos neg neutral 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 average attention weight neutral tokens, and calculated the average weight for each token. next we plot in figure 5 the distribution of such average attention weights for tokens of these three types separately. as we can observe, generally, the polarity tokens are more likely to have larger attention weights than the neutral tokens. however, the positive tokens seemed to receive lower scores than the negative tokens in terms of the attention weights. this is consistent with the attention scores shown in figure 2d : the attention scores of the positive tokens were generally lower than those of the negative tokens. meanwhile, we could see that there were some outliers of large weights for the neutral tokens (circles that appear outside the boxes are outliers). we looked into the case, it was due to that all of the three tokens in the short instance "is this progress" had negative attention scores, and the last token "progress" somehow had a relatively larger one, making its corresponding attention weight the largest amongst the three. this can be explained by the fact that attention weights only capture relative significance of tokens within a local context. these empirical results support our analysis as well as our belief on the significance of the attention scores. when certain hyperparameters are properly set, the attention mechanism tends to assign larger attention scores to those tokens which have strong association with instances of a specific label. meanwhile, the polarity scores for such tokens tend to yield large absolute values (of possibly different signs, depending on the polarity of the tokens), which will be helpful when predicting instance labels. by contrast, neutral tokens that appeared evenly across instances of different labels are likely assigned small attention scores and polarity scores, making them relatively less influential. ## conclusions in this work, we focused on understanding the underlying factors that may influence the attention mechanism, and proposed to examine attention scores -a global measurement of significance of word tokens. we focused on binary classification models with dot-product attention, and analyzed through a gradient descent based learning framework the behavior of attention scores and polarity scores -another quantity that we defined and proposed to examine. through the analysis we found that both quantities play important roles in the learning and prediction process and examining both of them in an integrated manner allows us to better understand the underlying workings of an attention based model. our analysis also revealed factors that may impact the interpretability of the attention mechanism, providing understandings on why the model may still be robust even under scenarios where the attention scores appear to be less interpretable. the empirical results of experiments on various real datasets supported our analysis. we also extended to and empirically examined additive attention, multi-label classification and models with an affine input layer, and observed similar behaviors. there are some future directions that are worth exploring. specifically, we can further examine the influence of using pre-trained word embeddingswhether similar words can help each other boost their polarity and attention scores. moreover, we can also examine the influence of using deep contextualized input encoders such as elmo @xcite or bert @xcite .