Emotion Enriched Retrofitted Word Embeddings (2022)

Abstract: Word embeddings learned using the distributional hypothesis (e.g., GloVe, Word2vec) are good at encoding various lexical-semantic relations. However, they do not capture the emotion aspects of words. We present a novel retrofitting method for updating the vectors of emotion bearing words like fun, offence, angry, etc. The retrofitted embeddings achieve better inter-cluster and intra-cluster distance for words having the same emotions, e.g., the joy cluster containing words like fun, happiness, etc., and the anger cluster with words like offence, rage, etc., as evaluated through different cluster quality metrics. For the downstream tasks on sentiment analysis and sarcasm detection, simple classification models, such as SVM and Attention Net, learned using our retrofitted embeddings perform better than their pre-trained counterparts (about 1.5 % improvement in F1-score) as well as other benchmarks. Furthermore, the difference in performance is more pronounced in the limited data setting.

## introduction in this work, we present a novel retrofitting method to learn emotion enriched embeddings. for knowledge, it relies on word-level emotion annotations available in the nrc word-emotion association lexicon (known as emolex). the central idea is: if words w a and w p are associated with the same emotion category t and word w n is not associated with t, then w a is semantically closer to w p than w n in the context of the emotion category t. this can be stated as an inequality constraint: sim t (w a , w p ) > sim t (w a , w n ). such emotion inequality constraints containing word triplets (w a , w p , w n ) are generated for all emotion categories present in emolex. we use these constraints as training data to learn a non-linear transformation function that maps original word vectors to a vector space respecting these constraints. the transformation function is learned in a similarity metric learning setting using a multi-layer feedforward network. the embeddings retrofitted using our method achieve better clustering for emotion bearing words. for the downstream tasks on sentiment analysis and sarcasm detection, they perform better than their pre-trained counterparts and other benchmarks, with significant gains in limited data setting. the main contributions of this work are: ## constraints from nrc emolex a large body of work has focussed on understanding and modelling human emotions. for instance, plutchik's wheel of emotions @xcite , ekman's model @xcite ), parrot's tree-structured emotions @xcite , etc. the model proposed by plutchik arranges emotions in circles with the length of radius indicating the intensity of emotions. it proposes eight basic or primary emotions: joy, trust, fear, surprise, sadness, disgust, anger, and anticipation. various lexical resources have been proposed in the literature to capture the emotion aspect of words, e.g. @xcite . in this work, we focus on nrc emolex (mohammad and turney, 2013). it contains a list of english words and their associations with plutchik's eight basic emotions @xcite . since words are ambiguous in their meaning and may evoke multiple emotions, each word in emolex has been associated with a set of emotions. for example, playful is associated with three emotion categories: trust, surprise and joy. we obtain a set of inequality constraints from emolex in the form of triplets. each triplet contains three words (w a , w p , w n ) in which we refer to w a , w p and w n as the anchor, positive and negative words, respectively. the corresponding inequality constraint is: similarity between w a and w p shall be greater than the similarity between w a and w n , by at least a margin m. the margin m is set in the range [0, 2] corresponding to a minimum versus maximum separation on cosine distance. for example, consider the following word-emotion pairs in emolex: (lonely, sadness), (playful, joy), and (sorrow, sadness). with lonely as the anchor word, sorrow can be considered the positive word since both these words belong to the same emotion category sadness. the word playful is then considered as a negative word since it is annotated with a different emotion category joy. this gives rise to the following constraint in the context of sadness category: sim sadness (lonely, sorrow) > sim sadness (lonely, playf ul) + m sadness . such constraints are obtained in the context of all the eight emotion categories by considering each word from the corresponding emotion category as the potential anchor and then generating the positive and negative words. ## retrofitting method our goal is to learn a transformation function that maps pre-trained word embeddings to a vector space that respects the emotion inequality constraints. as explained earlier, an inequality constraint is created using a word triplet (w a , w p , w n ) and the corresponding emotion category-specific margin m. thus, a natural way to create training data for this task is to generate a set of four tuples (w a , w p , w n , m) using all possible inequality constraints. however, training data generation using all inequality constraints has a drawback. we first explain this drawback and ways to mitigate it, followed by our retrofitting model to learn the transformation function. ## experimental results to evaluate our retrofitting method, we experimented with 300-dimensional pre-trained embeddings in glove foot_0 @xcite and word2vec foot_1 @xcite . due to space constraints, we discuss only glove results here (word2vec results are present in appendix b). as explained earlier, we used triplet constraints extracted from emolex to learn retrofitted embeddings. we refer to our method as retriplet hereafter. although we report the complete hyperparameter grid search details in appendix a, the hyper-parameter λ v for the vector space preservation loss in eq. 2 needs special attention. setting the right value for λ v is extremely important to learn a meaningful retrofitting model. if we set it very high, retriplet may not focus on the inequality constraints in triplet loss, thereby learning retrofitted embeddings nearly identical to their pretrained version. conversely, a low value of λ v may produce embeddings that largely satisfy emotion constraints but may not preserve the topology of input vector space, possibly leading to degraded performance on downstream tasks. to account for this trade-off, we devise the following scheme and select two configurations: (1) we use adjusted rand index (ari, a clustering evaluation metric, described in section 4.1) to measure the quality of retrofitted embeddings and select the configuration that gives the highest value for ari (referred as retripletg); (2) we compute the average cosine distance between pre-trained and retrofitted embeddings for words in emolex and filter configurations having @xmath0.15. we then choose the configuration with the highest ari from the filtered list (referred as retripletgbal). retrofitting approaches proposed in the literature use attract and repel constraints, extracted from wordnet, paraphrase database, etc., to update pretrained embeddings. the attract constraints pull similar (e.g., synonyms, hypernyms, etc.) word pairs close together. while the repel constraints push non-similar (e.g., antonyms) word pairs away from each other. we compare retriplet with the following, counterfit @xcite : it defines the loss function as a weighted sum of terms that brings attract word pairs closer and pushes repel word pairs apart. it also includes a vector space regularization term. attract-repel (ar) @xcite : the counterfit method updates embeddings of attract and repel words without considering their relations to other words. ar addresses this problem by performing context-sensitive vector updates. for each word in attract pairs, it finds the closest (in terms of cosine distance) in-batch word to generate negative examples (conversely farthest for repel words). it then uses these negative examples to form a hinge loss function for context-sensitive updates. post-specialization: the methods described above locally update vectors of only those words that are present in constraints (i.e., seen words), whereas vectors for all other words remain intact. to address this, post-specialization methods use retrofitted embeddings of seen words to learn a global specialization function which then updates vectors of unseen words. we use the generative adversarial network architecture proposed by @xcite for post specialization with ar as the local method (referred to as ar+ps). we also learn emotion enriched embeddings us-ing the methods described above by extracting attract and repel constraints from emolex. two words annotated with the same emotion category in emolex are added to the attract set, e.g. (angry, offence) since both angry and offence are marked with the anger category. in contrast, two words, when annotated with different emotion categories, are added to the repel set, e.g. (fun, miserable) since fun is marked with joy and miserable with sadness. the generated attract and repel sets are then used to learn retrofitted embeddings. they are referred by appending +el to the retrofitting method, e.g., ar+el for embeddings retrofitted using ar with emolex constraints. we also compare our method with the following emotion enriched embeddings: (1) ewe @xcite : it first creates noisy emotion labelled data using distant supervision and then applies recurrent neural network to learn emotion embeddings; (2) aff2vec @xcite : it appends valence (v), arousal (a) and dominance (d) values of words as present in warriner's vad lexicon @xcite to the counterfitted glove embeddings, resulting in 303-dimensional affective embeddings; (3) eearmin @xcite : it applies counterfit method directly on the (word, emotion) pairs in emolex; (4) sentiembs @xcite : embeddings refined for sentiment using valence values present in warriner's lexicon. ## related work large language models with contextualized word embeddings (e.g., bert and its variants) have lately received a lot of attention in the nlp community. nonetheless, their static counterparts are still actively explored, e.g., combining static and contextualized embeddings to improve end-tasks @xcite , inducing knowledge bases @xcite , bilingual lexicon induction @xcite , etc. in this work, we focus on static word embeddings that are learned primarily using the distributional hypothesis. a major limitation with these embeddings is that they do not differentiate semantic similarity from other types of relatedness @xcite . this problem is addressed by borrowing semantic relations from resources such as wordnet, paraphrase database, etc., in the form of constraints. these constraints are then used by joint specialization @xcite or retrofitting models @xcite @xcite to improve word embeddings. these models, however, focus mainly on synonymy, antonymy, and hypernymy relations. recently, a few attempts, such as aff2vec from @xcite and emotion embeddings from @xcite , incorporate knowledge present in affective lexicons to learn emotion enriched embeddings. the contrastive learning approach similar to our work has recently been applied to learn transformer based sentence embeddings in sbert @xcite and zero-shot image classification in clip @xcite . however, these methods are not specialized to learning emotion enriched embeddings. there is a large body of work that focuses on learning task-specific affective embeddings. these methods first use distant supervision to create a noisy labelled dataset and then use it to update word embeddings or learn them from scratch. for instance, sentiment-aware embeddings using tweet data @xcite ; affective embeddings using tweet emojis @xcite ; emotion enriched embeddings using product reviews data @xcite . since embeddings learned from these methods are tied to the dataset used for distant supervision, they may not work well for other related affective end-tasks. moreover, they are not very accurate due to noisy labelling. the emotion-enriched embeddings learned by our method are not only accurate compared to the methods described above, as evident from the clustering experiments, they also work well on the related affective end-tasks. ## summary and future work we present a novel retrofitting method to learn emotion enriched embeddings using triplet constraints from emolex. these constraints are used as training data to learn a retrofitting function in a similarity metric learning setting. the embeddings learned by our method perform better than their pre-trained counterparts and other benchmarks in both intrinsic clustering evaluation and the extrinsic downstream tasks in sentiment analysis and sarcasm detection. as future work, we plan to extend our triplet constraint-based approach to other resources such as vad lexicon @xcite . we also plan to develop a similar approach for contextualized word embeddings.