Enhancing Self-Attention via Knowledge Fusion: Deriving Sentiment Lexical Attention from Semantic-Polarity Scores (2024)

Abstract: In recent years, pre-trained language models have demonstrated exceptional performance across various natural language processing (NLP) tasks. One fundamental component of these models is the self-attention mechanism, which has played a vital role in capturing meaningful relationships between tokens. However, a question still remains as to whether injecting lexical features into the self-attention mechanism can further enhance the understanding and performance of language models. This paper presents a novel approach for injecting semantic-polarity knowledge, referred to as Sentiment Lexical Attention, directly into the self-attention mechanism of Transformer-based models. The primary goal is to improve performance on sentiment classification task. Our approach involves consistently injecting Sentiment Lexical Attention derived from the lexicon corpus into the attention scores throughout the training process. We empirically evaluate our method on the NSMC sentiment classification benchmark, showcasing significant performance improvements and achieving state-of-the-art results. Furthermore, our approach demonstrates robustness and effectiveness in out-of-domain tasks, indicating its potential for broad applicability. Additionally, we analyze the impact of Sentiment Lexical Attention on the view of the CLS token’s attention distribution. Our method offers a fresh perspective on synergizing lexical features and attention scores, thereby encouraging further investigations in the realm of knowledge injection utilizing the lexical features.

## introduction in recent years, pre-trained language models such as bert @xcite , xlnet @xcite , bart @xcite , and gpt-3 @xcite have demonstrated remarkable performance across various downstream tasks in nlp. these language models (lms) are characterized by a vast number of trainable parameters, which many researchers believe encode valuable knowledge during the processing of contextualized token embeddings @xcite . among these parameters, self-attention mechanisms play a crucial role and are widely considered as the foundation of nearly all language models. many studies have led to performance improvements by attempting to inject knowledge into self-attention, based on the understanding that it learns based on relationships between tokens @xcite @xcite . however, an important question remains: can we leverage the sentiment lexical features to enhance the self-attention mechanism and gain a deeper understanding of the relationships between semantically meaningful tokens? many studies have investigated the methods of knowledge fusion on lms to enhance performance in natural language understanding tasks @xcite @xcite . knowledge injection techniques can be applied to any part of the lms @xcite . among the many methods, we introduce a method to convert lexical sentiment features into a computable matrix (sentiment lexical attention), which is then induced into a linear combination with the self-attention. we denote this fused attention mechanism as sentiment-fused attention (figure 1 ). for injecting sentiment-fused attention well on attention score, we partially follow @xcite 's way, which proposes a method of guiding the attention output by injecting similarity knowledge into the attention score. in section 3, we suggest our novel approach of injection, pointing out @xcite 's injection methods might have the potential to distort the relationships of tokens. furthermore, the process of extracting sentiment lexical features from the polarity score of the lexicon corpus and deriving them into a fusible matrix is elaborately described in section 3. we believe our method of injection is meticulously formulated to augment the weights between tokens with similar sentiment features. in section 4, we substantiate the effectiveness of our injection method through experiments, simultaneously empirically demonstrating that it yields stable performance improvements, unlike @xcite 's approach. to the best of our knowledge, our results represent the stateof-the-art on the naver sentiment movie corpus (nsmc) foot_0 , which is widely regarded as the leading benchmark for sentiment analysis in the korean language. additionally, experimental observations reveal the effectiveness of sentiment lexical features in out-of-domain tasks. in section 6, we investigate the impact of sentiment-fused attention on attention by statistically examining the attention dynamics of the cls token (serving as the classifier) and demonstrate through analysis that it exerts significant influence. the main contribution of our work are as follows: ## related work previous research has extensively investigated the injection of knowledge into self-attention based language models to augment its language representation prowess @xcite . in this chapter, we introduce prior research on knowledge graphbased approaches, which are most commonly utilized for knowledge injection, and discuss how knowledge integration has been approached from the perspective of lexical semantics. finally, we justify the validity of our research by introducing prior studies related to self-attention distributions. ## direct injection of sentiment lexical attention into self-attention in this investigation, we enhance the existing self-attention mechanism by embedding sentiment lexical attention within its attention matrix, thereby integrating sentiment-related connections among tokens. sentiment lexical attention is conceptualized through the quantification of semanticpolarity similarity among token pairs, established via the dot product computation of their context polarity vectors. this process engenders a semanticpolarity similarity matrix that meticulously delineates the sentiment linkages inherent in tokens within a specific input sequence, ensuring a nuanced comprehension of these interrelations. notably, a pronounced amplification of polarity similarities is observed among tokens sharing analogous sentiment properties, with the similarity values delineated within a spectrum ranging from 0 to 1. by leveraging this semantic-polarity similarity as sentiment lexical attention, we could directly inject this information into attention scores. this methodology enables us to potentially refine the attention mechanism by injecting sentimentassociated values between tokens. consequently, this process facilitates the generation of a more informed representation of the sentiment relationships within sentences. ## dissecting the impact of σ ij on attention dynamics: an in-depth analysis centered on the cls token in this section, we investigate the differences in standard deviation between the baseline model and the att last + √ d k model concerning the cls token at each layer. our approach involves the direct injection of sentiment lexical attention into the attention scores. we hypothesize that this injection of knowledge will lead to alterations in the relationship centered on the cls token, which serves as the representative vector for the subsequent classifier. to test this hypothesis, we conduct an analysis of the standard deviation of attention scores surrounding the cls token at each layer, aiming to understand the impact of σ ij . for statistical analysis, we examine the standard deviation of attention scores between the cls token and other tokens in both the baseline model and the att last + √ d k model. we conduct this analysis layer-by-layer while processing the 50k test dataset from the nsmc dataset. by comparing the standard deviation of attention scores, we aim to understand how the attention patterns of the cls token change when sentiment lexical attention is incorporated into the model. this analysis provides insights into the impact of knowledge injection on the attention mechanism and its effect on the relationship between the cls token and other tokens. figure 5 demonstrates that the deviation between the baseline models and the att last + √ d k model, specifically regarding the cls tokens, primarily manifests in the alterations in the distribution of attention scores between the cls token and other tokens. the presence of such disparities between models that differ solely based on the injection of σ ij in their training processes strongly suggests a significant influence of σ ij on the dispersion of attention scores. interestingly, the distribution of attention scores from other tokens towards the cls token remains mostly unchanged. these findings can be attributed to the fact that the cls token does not derive its context polarity λ(x) from λ(v), resulting in minimal differences in the attention weights towards the cls token compared to the baseline models. on the other hand, tokens other than the cls token, influenced by λ(v), consistently induce modifications in the attention score distribution throughout the training process, which likely affects the final attention distribution of the model. through this analysis, we propose that these shifts in attention distribution serve as the primary catalyst for the performance alterations depicted in tables 1 and 2 . ## discussion in this paper, we have introduced a novel approach for enhancing the self-attention mechanism of transformer-based models through the injection of sentiment lexical attention, derived from semantic-polarity scores. our results demonstrate significant improvements in sentiment classification, particularly in the korean language context. however, the applicability and challenges of this method across different tasks and languages, as well as its technical novelty, warrant further discussion. ## conclusion in our study, we introduced a new approach to inject sentiment knowledge into the self-attention mechanism of transformer-based models. this approach yielded significant improvements, particularly in korean sentiment classification benchmark, where we achieved a new state-of-the-art performance. moreover, the promising results obtained across various out-of-domain tasks highlighted the general applicability of our method. although the observed performance variations were task-and model-dependent, they underscored the substantial potential of incorporating human-derived knowledge into transformer-based language models. furthermore, in our examination of the cls token, we could ascertain the direct impact of knowledge injection on the layer-wise attention distribution. the approach presented in this study opens the door for further exploration of effective techniques for injecting human knowledge into language models.