Structured Self-Attention Weights Encode Semantics in Sentiment Analysis (2020)

Abstract: Neural attention, especially the self-attention made popular by the Transformer, has become the workhorse of state-of-the-art natural language processing (NLP) models. Very recent work suggests that the self-attention in the Transformer encodes syntactic information; Here, we show that self-attention scores encode semantics by considering sentiment analysis tasks. In contrast to gradient-based feature attribution methods, we propose a simple and effective Layer-wise Attention Tracing (LAT) method to analyze structured attention weights. We apply our method to Transformer models trained on two tasks that have surface dissimilarities, but share common semantics—sentiment analysis of movie reviews and time-series valence prediction in life story narratives. Across both tasks, words with high aggregated attention weights were rich in emotional semantics, as quantitatively validated by an emotion lexicon labeled by human annotators. Our results show that structured attention weights encode rich semantics in sentiment analysis, and match human interpretations of semantics.

## introduction in recent years, variants of neural network attention mechanisms such as local attention @xcite and self-attention in the transformer @xcite have become the de facto go-to neural models for a variety of nlp tasks including machine translation @xcite , syntactic parsing @xcite , and language modeling @xcite . attention has brought about increased performance gains, but what do these values 'mean'? previous studies have visualized and shown how learnt attention contributes to decisions in tasks like natural language inference and aspect-level sentiment @xcite @xcite . recent studies on the transformer @xcite have demonstrated that attention-based representations encode syntactic information @xcite such as anaphora @xcite , partsof-speech @xcite and dependencies @xcite @xcite . other researchers have also done very recent extensive analyses on self-attention, by, for example, implementing gradient-based layer-wise relevance propagation (lrp) method on the transformer @xcite to study attributions of graident-scores to heads, or graph-based aggregation method to visualize attention flows @xcite . these very recent works have not looked at whether the structured attention weights themselves aggregate on tokens with strong semantic meaning in tasks such as sentiment analysis. thus, it is still unclear if the attention on input words may actually encode semantic information relevant to the task. in this paper, we were interested in extending previous studies on attention and syntax further, by probing the structured attention weights and studying whether these weights encode task-relevant semantic information. in contrast to gradient-based attribution methods @xcite , we were explicitly interested in probing learnt attention weights rather than analyzing gradients. to do this, we propose a layer-wise attention tracing (lat) method to aggregate the structured attention weights learnt by self-attention layers onto input tokens. we show that these attention scores on input tokens correlate with an external measure of semantics across two tasks: a sentiment analysis task on a movie review dataset, and an emotion understanding task on a life stories narrative dataset. these tasks differ in structure (single-example classification vs. timeseries regression), and in domain (movie reviews vs. daily life events), but should share the same semantics, in that the same words should be important in both tasks. we propose a method of external validation of the semantics of these tasks, using emotion lexicons. we find evidence for the hypothesis that if self-attention mechanisms can learn emotion semantics, then lat-calculated attention scores should be higher for words that have stronger emotional semantic meaning. ## attention-based model architecture we use an encoder-decoder architecture as shown in fig. 1 . our encoder is identical to the encoder of the transformer @xcite , with an additional local attention layer @xcite . our decoder is task-specific: a simple multilayer perceptron (mlp) for the classification task, and a lstm followed by a mlp for the time-series prediction task. self-attention layers. the encoder is identical to the original transformer encoder and consists of a series of stacked self-attention layers. each layer contains a multi-head self-attention layer, followed by an element-wise feed forward layer and residual connections. following @xcite , we @xmath0, and a hidden dimension @xmath1. we briefly recap the transformer equations, to better illustrate our lat method, which traces attention back through the layers. for a given selfattention @xmath2, ], we denote the input to using @xmath3, which represents tokens, each embedded using a -dimensional embedding. we keep the same input embedding size for all layers. the first layer takes as input the word tokens. a selfattention layer learns a set of query, key and values matrices that are indexed by (i.e., weights are not shared across layers). formally, these matrices are produced in parallel: each head learns a self-attention matrix s( ) ℎ using the scaled inner product of q ℎ and k ℎ followed code is available at https://github.com/ frankaging/lat_for_transformer by a softmax operation. the self-attention matrix s( ) ℎ is then multiplied by v ℎ to produce z ℎ : next, we concatenate z ℎ from each head ℎ to produce the output of layer (i.e., the input to layer + 1) x +1 , ## layer-wise attention tracing to study whether structured attention weights encode semantics, we propose a tracing method, layerwise attention tracing (lat), to trace the attention 'paid' to input tokens (i.e. words) through the selfattention layers in our encoder. lat, illustrated in fig. 2 , involves three main steps. first, starting from the local attention layer and a fixed "quantity" of attention, we distribute attention weights back to z ℎ , the last self-attention layer of each head ℎ ∈ [1, ]. second, we trace the attention back through each self-attention @xmath4, ]. third, from the first layer of each head, we trace the attention back onto each token in the input sequence, by accumulating attention scores from each head to the corresponding position. we do not consider the decoder in lat, as the mlp and lstm layers in the decoder do not modify attention. furthermore, we specifically ignore the feedforward layers and residual connections in the encoder, as we were interested in the attention , not the neural activations they modify-this is our main differentiation from gradient-based or relevance-based work @xcite , and we note another recent paper @xcite ) that made the same assumptions. ## related work there has been extensive debate over what attention mechanisms learn. on the one hand, researchers have developed methods to probe learnt self-attention in transformer-based models, and show that attention scores learnt by models like bert encode syntactic information like partsof-speech @xcite , dependencies @xcite , anaphora @xcite and other parts of the traditional nlp pipeline @xcite . these studies collectively suggest that self-attention mechanisms learn to encode syntactic information, which led us to propose the current work on whether selfattention can similarly learn to encode semantics. on the other hand, there are also other papers questioning the interpretations the field has placed on attention. these researchers show that attention weights have a low correlation with gradient-based measures of importance @xcite @xcite . more recent analysis suggest that in certain regimes for the transformer (i.e., sequence length greater than attention head dimension ), attention distributions are non-identifiable, posing problems for interpretability @xcite . in our work, we provide a method that can trace attention scores in transformers to the input tokens, and show with both qualitative and quantitative evidence that these scores are semantically meaningful. beyond attention-based studies, there have been numerous studies that proposed gradient-based attribution analyses @xcite @xcite and layer-wise relevance propagation @xcite @xcite . most related to the current work is @xcite , who extended layer-wise relevance propagation to the transformer to examine the contribution of individual heads to the final decision. in parallel, @xcite recently proposed a method to roll-out structured attention weights inside the transformer model, which is similar to our lat method we propose here, although we provided more analysis via an external validation using external knowledge. we sought to investigate the attention accumulated onto individual input tokens using attention tracing, in a more similar manner to @xcite for syntax or how @xcite looked at the attention paid to other words. we also calculate a gradient-based score (see eqn. 13) to contrast our attention results with, and though these two scores are correlated (see footnote 6), they behave differently in our analyses. we note that our models do not fall into this regime. ## datasets to show that our interpretation methods can generalize across different types of datasets, we apply our method to two tasks with different characteristics, namely, sentiment classification of movie reviews on the stanford sentiment treebank (sst), and time-series valence regression over long sequences narrative stories on the stanford emotional narratives dataset (send). ## discussion in this work, we analyzed whether structured attention weights encode semantics in sentiment analysis tasks, using our proposed probing method lat to trace attention through multiple layers in the transformer. we demonstrated that the accumulated attention scores tended to favor words with greater semantic meaning, in this case, emotional meaning. we applied lat to two tasks having similar semantics, and show that our results generalize across both tasks/domains. we validated our results quantitatively with an emotion lexicon, and showed that our attention scores are highest for both highly positive and highly negative words-our a priori hypothesis for the quadratic, "u-shaped" relationship. we also found some evidence for specialization of heads to emotional meaning. although it may seem that our attention tracing is "incomplete" as it does not take into account the feed-forward layers and residual connections, by contrast, this quadratic relationship was not shown by pure gradient-based importance, which suggests that there may be some utility to looking only at attention. we believe that attention in its various forms @xcite are not only effective for performance, but may also provide that is, when calculating a 1 ℎ (very positive), we exclude sequences that do not contain at least 1 very positive word. interpretable explanations of model behaviour. it may not happen with today's implementations; we may need to engineer inductive biases to constrain attention mechanisms in order to address issues of identifiability that @xcite and others have pointed out. and perhaps, attention should not be interpreted like gradient-based measures (see fig. 5 ). this debate is not yet resolved, and we hope our contributions will be useful in informing future work on this topic.