Affective Retrofitted Word Embeddings (2022)

Abstract: Word embeddings learned using the distributional hypothesis (e.g., GloVe, Word2vec) do not capture the affective dimensions of valence, arousal, and dominance, which are present inherently in words. We present a novel retrofitting method for updating embeddings of words for their affective meaning. It learns a non-linear transformation function that maps pre-trained embeddings to an affective vector space, in a representation learning setting. We investigate word embeddings for their capacity to cluster emotion-bearing words. The affective embeddings learned by our method achieve better inter-cluster and intra-cluster distance for words having the same emotions, as evaluated through different cluster quality metrics. For the downstream tasks on sentiment analysis and sarcasm detection, simple classification models, viz. SVM and Attention Net, learned using our affective embeddings perform better than their pre-trained counterparts (more than 1.5% improvement in F1-score) and other benchmarks. Furthermore, the difference in performance is more pronounced in limited data setting.

## introduction affect refers to the experience of a feeling or emotion @xcite . this definition broadly encompasses sentiment, emotion, personality, and mood. incorporating these affective aspects in text analysis can significantly benefit numerous nlp applications, including sentiment analysis, sarcasm detection, opinion mining, empathetic agents, etc. words, being the smallest meaningful constructs in a language, have been the primary focus area for affect analysis in literature. the affective meaning of a word can be represented primarily using: (1) discrete affective labels such as joy, happiness, anger, etc., notable models include plutchik's wheel of emotions @xcite , @xcite , etc.; (2) dimensional models such as valence-arousal-dominance (vad) model @xcite , evaluation-potency-activity (epa) model @xcite , etc. that represent human affects in a continuous space. in this work, we focus on dimensional models since they capture more fine-grained information compared to the discrete models and are more expressive @xcite . the dimensional model in vad represents a word and its affective meaning as a point in a 3-dimensional space that consists of valence (degree of pleasure or displeasure), arousal (degree of excitement or calmness), and dominance (degree of control or submission). while pre-trained embeddings are good at capturing various lexico-semantic relations, do they encode the affective meaning of words? for example, consider violate, a word having low valence and high arousal. it is well known in the community that the embeddings learned using the distributional hypothesis @xcite ) mix semantic similarity with other types of semantic relatedness @xcite . for instance, though opposite in meaning, both cheap and expensive have similar embeddings since they occur in nearly identical contexts. this problem has been addressed by first borrowing semantic relations from knowledge sources such as wordnet, paraphrase database, etc., in the form of constraints and then using these constraints to learn joint specialization @xcite or retrofitting @xcite pre-trained word2vec vadprojwbal violate (↓v;↑a) contravene, violation, abide, prohibit, adhere, forbid, comply, contravention, obey, stipulate contravene, prohibit, endanger, forbid, restrict, violation, oppose, abide, offend, discriminate bombard (↓v;↑a) barrage, overwhelm, saturate, zap, invade, terrorize, ignore, hurl, swarming, scour overwhelm, terrorize, saturate, hurl, frighten, obliterate, gobble, zap, invade, unleash conquer (↑a;↑d) conquering, vanquish, overcome, liberate, annihilate, conquest, banish, unite, outwit, confront conquering, vanquish, liberate, overcome, annihilate, unleash, unite, outwit, confront, wrest in this work, we present a simple yet effective retrofitting approach to learn vad-enriched affective embeddings. for knowledge, it relies on the real-valued valence, arousal, and dominance scores available in the nrc vad lexicon @xcite . we hypothesize that when we map pretrained embeddings to a vector space that is conducive to predicting vad scores, the mapped vectors acquire affective meaning, resulting in affective embeddings. we design the mapping function as a non-linear transformation using a multi-layer feed-forward neural network. given an input word, we first compute its affective embedding using the mapping function. the affective embedding is then linearly projected to a 3-dimensional vector space corresponding to the vad dimensions. the scores present in the vad lexicon are used to jointly learn both the mapping function as well as the linear vad projection. the affective embeddings learned using our method achieve better clustering for emotion bearing words. for downstream tasks on sentiment analysis and sarcasm detection, they perform better than their pre-trained counterparts and other benchmarks, with significant gains in limited data setting. the main contributions of this work are: ## nrc vad lexicon various lexical resources have been proposed in the literature to capture the affective meaning of words using dimensional models, e.g., anew @xcite ), warriner's lexicon @xcite , etc. in this work, we leverage the knowledge present in the vad lexicon @xcite to learn affective embeddings. the lexicon provides real-valued scores in the range [0, 1] for valence (v), arousal (a), and dominance (d) (0=low; 1=high) for more than 20,000 english words. ## retrofitting method our goal is to learn a non-linear transformation function that maps pre-trained word embeddings to a vector space that encodes the affective meaning of words. the first question that arises here is: how do we measure or quantify the degree of affect content in a given vector space? we argue that it should be easy to extract the affective meaning of words from such a vector space. in fact, we hypothesize and show (refer section 4) that a simple linear projection of word vectors from such a space to a 3-dimensional vad space accurately extracts or predicts valence, arousal, and dominance scores of words. therefore, we treat the linear projection to the vad space as our objective criteria to learn the transformation function. to this end, the valence, arousal, and dominance scores present in the vad lexicon provide the required training data. figure 1 shows the overall architecture for learning our retrofitting model for affective embeddings. 1. training data generation: a training example in our model consists of a word and its vad scores. generally, the number of words with high affect scores, either positive or negative, is limited in a language. conversely, a large number of words have average affect scores. figure 2 shows the histograms of vad scores for the words in the vad lexicon, depicting this language property. regression models learned for target variables with such skewed distribution become biased, generally leading to better performance for common values than rare cases. however, the words that are referred more often to stress emotional or affective aspects in human communication generally have either positive or negative affect content as opposed to the average score. for example, consider words such as {happy, nightmare, weak, etc.}, and {indulgence, pretend, lease, etc.}. the former set contains words that exhibit affective aspects, whereas the latter contains words with minimal or no affective content. since the words having extreme or rare vad scores are of particular importance in our case, this imbalance in affect scores needs to be taken into account while learning our retrofitting model. we employ a sample weighting approach with cost-sensitive learning to address the imbalanced regression problem described above. specifically, sample weights are assigned to each word w i in the vad lexicon such that the words with high/low affect scores get higher weights than those with average affect scores. we use the density-based weighting scheme (denseweight) proposed by @xcite to compute sample weights. the fol-lowing describes the process. ## experimental results to evaluate our method, we experimented with 300dimensional pre-trained embeddings in word2vec foot_0 @xcite and glove foot_1 @xcite . due to space constraints, we discuss only word2vec results here (refer appendix b for glove). the complete hyper-parameter grid search details, computational cost, etc. are detailed in appendix a. as discussed earlier, the transformation function that maps pre-trained word embeddings to an affective vector space is learned in a regression setting using the loss function in eq. 1. this loss function contains two contrasting terms, viz. vad regression loss (l vad ) and vector space preservation loss (l v ). the hyper-parameter λ v provides a knob to balance these contrasting terms and needs to be set at the right value to learn a meaningful transformation function. setting a very high value for λ v will make our model ignore the affective content of words, thereby learning retrofitted embeddings nearly identical to their pre-trained version. conversely, a low value of λ v may produce embeddings that predominantly contain affective meaning at the expense of forgetting lexico-semantic rela-tions present in the pre-trained vector space, possibly leading to degraded performance on end-tasks. to select the best hyper-parameter configuration, we conduct two experiments. (1) we directly select the configuration that gives the least mse 3 in predicting vad scores (referred as vadprojw) (2) we first compute the mean cosine distance between the pre-trained and affective embeddings of words and select configurations with a @xmath0.15. we then choose the best configuration (with the least mse in vad prediction) amongst the filtered list (referred as vadprojwbal). ## related work word embeddings built using the distributional hypothesis have been studied extensively in the literature for the types of semantic relations they encode. it has been observed that they mix semantic similarity with other types of relatedness @xcite , potentially leading to degraded end-task performance. various joint learning @xcite or retrofitting @xcite @xcite models address this problem by leveraging semantic relations from resources such as wordnet, paraphrase database, etc. however, they focus mainly on synonymy, antonymy, and hypernymy relations. to inject affective meaning into word embeddings, a few attempts @xcite have recently used resources such as emolex @xcite and affect intensity lexicon (mohammad, 2018b) that cater to discrete affective models. these methods, however, are limited by the coarse-grained affect labelling and lack finer affective interpretations. lately, @xcite and @xcite have used dimensional model resources such as warriner's lexicon @xcite and vad lexicon @xcite to encode fine-grained affective meaning. different from affect, there also exist lexicons that can be used to ground the semantic meaning of affect bearing words into other modalities. for example, colors in the nrc word-color association (e.g. danger -red) lexicon @xcite ; perceptual modalities and action effectors in lancaster sensorimotor norms @xcite ; robot state behavior @xcite , etc. a large body of work focuses on learning taskspecific affective embeddings. these methods first generate a noisy labelled dataset using distant supervision and then use it to update word embeddings or learn them from scratch. notable works include sentiment-aware embeddings @xcite using tweet data, affective embeddings @xcite using tweet emojis, emotionenriched embeddings @xcite using product reviews, etc. however, the embeddings learned from these methods are customized with dataset-specific nuances and might also model noise inherently present due to distant supervision. due to this, they do not generalize well across other related tasks. the affective embeddings learned by our retrofitting method are not only accurate compared to the methods described above, as evident from the clustering experiments, but also work well on the related affective end-tasks. ## summary and future work we present a simple yet effective retrofitting method to learn affective embeddings using the nrc vad lexicon. the affect scores in the lexicon are used as training data to learn a transformation function in a representation learning setting that maps pre-trained embeddings to an affective vector space. the embeddings learned by our method perform better than their pre-trained version and other benchmarks in both the intrinsic task of clustering emotion-bearing words and the affective downstream tasks in sentiment analysis and sarcasm detection. we are currently extending our retrofitting approach to other affective resources such as affect intensity lexicon @xcite and emolex @xcite . we also plan to develop a similar approach for contextualized word embeddings. @xcite . learning sentiment-specific word embedding for twitter sentiment classification. in proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: long papers), pages 1555-1565, baltimore, maryland. association for computational linguistics. amy beth warriner, victor kuperman, and marc brysbaert. 2013. norms of valence, arousal, and dominance for 13,915 english lemmas. behavior research methods, 45:1191-1207. liang-chih yu, jin wang, k. robert lai, and xuejie zhang. 2017. refining word embeddings for sentiment analysis. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 534-539, copenhagen, denmark. association for computational linguistics. mo yu and mark dredze. 2014. improving lexical embeddings with semantic knowledge. in proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: short papers), pages 545-550, baltimore, maryland. association for computational linguistics.