Retrofitting of Pre-trained Emotion Words with VAD -dimensions and the P lutchik Emotions (2021)

Abstract: The word representations are based on distributional hypothesis according to which words that occur in the similar contexts, tend to have a similar meaning and appear closer in vector space. For example, the emotionally dissimilar words ”joy” and ”sadness” have higher cosine similarity. The existing pre-trained embedding models lack in emotional words interpretations. For creating our VAD-Emotion embeddings, we modify the pre-trained word embeddings with emotion information. This is a lexicons based approach that uses the Valence, Arousal and Dominance (VAD) values, and the Plutchik’s emotions to incorporate the emotion information in pre-trained word embeddings using post-training processing. This brings emotionally similar words nearer and emotionally dissimilar words away from each other in the proposed vector space. We demonstrate the performance of proposed embedding through NLP downstream task - Emotion Recognition.

## introduction an emotion is a feeling that characterizes the state of mind such as happiness, sadness, anger, fear and more. the emotions are classified using various taxonomies under the dimensional models and the psychological emotion models such as @xcite emotion model , @xcite emotion wheel, @xcite model which agree to a basic set of emotion with few changes. the plutchik emotion model represents the categorization of emotion words into 8 basic emotions : anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. the pad emotional state model @xcite is a 3-dimensional model that represents every emotion in valence (pleasure), arousal and dominance dimensions. emotion detection in the text is critical for a number of applications and services in diverse domains, including market research, customer-care, psychological healthcare, and intelligent tutoring systems and so on. @xcite . the automatic detection of emotions remains a challenging task till date as researchers may use different emotion models with different number and types of emotion categories. also, the emotions are subjective, hence creation of emotion related resources requires much time and effort. word embedding are distributed word representations where each word w in the vocabulary v is mapped into a dense, lowdimensional, continuous valued vector v w ϵr d . here d represents dimensions of the vector space model. most of the embeddings are modeled using the syntactic context of words which means words appearing in the similar contexts have the similar semantics and appear closer in the vector space @xcite , @xcite . as a consequence, emotionally opposite words, such as "joy" and "sorrow" occurring in similar contexts show higher cosine similarity. hence, said property does not fit in case of emotion words as 'joy' and 'sorrow' and many more similar and opposite emotion words too. we propose a model that modifies the pre-trained word embedding with emotion information using a post-training processing method. this is a lexicons based approach that uses the valence, arousal and dominance (vad) values, and the plutchik's emotions to incorporate the emotion information in the word embeddings. the main contributions of this paper includes the creation of emotion-fitted embedding to be used for nlp downstream tasks related to emotion analysis. we present average of cosine similarities for emotion words. the visualizations shown for nrc-emolex lexicons using for glove-300d embeddings, and retrofitted embeddings at both steps : vad-append embeddings and vad-emotion embeddings confirms the step-by-step clustering of similar emotion words. the accuracy for emotion recognition as downstream emotion task using the proposed embedding is mentioned as results. though clustering of emotion words takes place, the accuracy for emotion recognition task using proposed embedding is closer to baseline but can not outperform the same. the paper is organized as follows. section 2 discusses various approaches used by researchers for updating vector space models for specific set of nlp tasks. section 3 describes the proposed approach. experiment and setting are explained in section 4. we discuss the results and observation in section 5. ## related work the use of lexical semantic information (lexical resources), sentiment information, emotion information to improve distributional representations in respective area of nlp tasks is recent. methods like @xcite , @xcite , @xcite achieve improved representations by using training on unlabelled corpora, distant supervision and other techniques to gain relational knowledge to modify the prior or add a regularization term. such methods are known as 'pre-training methods', as they alter the training process for word representations. such methods may require a change in the loss function with training and may be computationally expensive. @xcite proposed a method for sentiment analysis, where they use external knowledge from sentiwordnet @xcite , extended anew lexicons @xcite with pre-trained word embeddings during joint parameter training to a cnn classifier using training data. @xcite proposed a distant supervision method for automatically labeling a large corpus of training data with fine-grained emotions; and the lstm model architectures for learning emotion-enriched word embedding from this training data. on the other hand post-training methods include external information to modify to the vanilla word representations such as word2vec, glove to name a few. retrofitting method @xcite , has used word relation knowledge from semantic lexicons (e.g. wordnet), to bring similar words closer in the retrofitted vector space. it injects antonym and synonym constraints to improve the existing word representations. @xcite presented post-training approach named as counter-fitting which injects antonym and synonym constraints into existing vector space representations in order to improve the vectors' capability for increase semantic similarity. aff2vec @xcite aims at incorporating affective information in word representations. they have used the warriner's vad lexicon @xcite to improve the strength in the antonym-synonym relationships of the words is incorporated to the word distribution space. the word similarity task and other sentiment analysis related tasks are performed to display the results. @xcite proposed a method, based on counterfitting approach @xcite that incorporates emotional information of words into the model. it uses an nrcemotional lexicon @xcite and the plutchik's model of basic emotions to prepare emotion constraints for fitting an emotional information into pre-trained word vectors. here, we propose a pipeline model to integrate valence, arousal and dominance information of emotion words and their basic emotion labels from lexicon set to create a new vector space on pre-trained word vectors using post-training (postprocessing) method. ## fitting vad values and emotion constraints into the word embedding this work aims at incorporating affect information in word representations. the figure -1 illustrates overview of proposed system for retrofitting of pretrained vector space. the emotion information is infused in two consecutive steps. the subsections 3.1 and 3.2 discuss the processing performed in step-1 and step-2 respectively. as shown in figure-1, at step-1, we append the valence, arousal and dominance (vad) values for the emotion lexicons to their respective word embedding to transform an original pre-trained vector space v into the vad-appended vector space v'. in step-2, we use emotion constraints to modify these vadappended embeddings further to achieve retrofitted vector space v". following subsections present the steps of modifications in the pre-trained word vector space v to v' and then to the final modified vector space v" from v'. nrc-vad lexicons : nrc-vad lexicon (mohammad, 2018) is a set of affect lexicons with 20007 english words. it contains real-valued scores for valence, arousal, and dominance (vad) on a scale of [0 -1] for each lexicon. ## experiments we have used the nrc-vad lexicons @xcite to create vad vectors and the nrc-emolex emotion lexicons @xcite for emotion constraints creation as they are labelled with plutchik's wheel of emotions @xcite along with positive and negative sentiment values. during experiments, glove-300d @xcite , fasttext-1m @xcite , and google word2vec @xcite are used as pre-trained input vector space and created the vad-append embeddings at step-1, vad-emotion word embeddings at step-2 respectively. we run stochastic gradient descent (sgd) for 20 epochs to achieve the final retrofitted embeddings with emotion information. we have compared performance of vad-emotion fitted embeddings with existing vector space as mentioned above for the task of finding average cosine similarity over nrc-emotion lexicons for similar and opposite emotions. also, have displayed accuracy results of emotion recognition task on isear dataset (isear) and the twitter emotion corpus (tec) dataset @xcite . ## results and discussion table-2 shows average cosine similarity between nrc-emolex lexicons and their respective emotions. higher the cosine similarity is better for the similar emotion words. for joy and trust emotions, vad-emotion embedding perform better than rest of them. table-3 shows average cosine similarity between nrc-emolex lexicons and the opposite emotions.the cosine similarity between lexicons and emotion should be as low as possible. the range for cosine similarity is [+1, -1] the figure -5 shows the clustering of similar emotion words together and opposite emotion words in opposite word clusters. table-4 shows accuracy of emotion recognition task using the pre-trained word embedding, and respective vad-append and vad-emotion embedding as input to a simple bilstm model. the table-4 reports 4-fold cross validation accuracy on the subset of isear dataset with 5.4k examples and tec dataset with 20k examples respectively used for training and testing. for the purpose of comparison, the emotion recognition task on the isear dataset was performed with the emotion-refined-embedding @xcite with glove-300d pretrained embedding which resulted as accuracy of 64.84%. the embeddings are computed with help of the code of @xcite which is available publicly. final proposed retrofitted embeddings. it can be observed that vad-emotion embeddings show the better cosine similarity among the emotionally similar words and less cosine similarity between emotionally dissimilar words than rest of them. this can be confirmed by looking at the clusters of emotion lexicons formed in figure -5. yet, the accuracy for emotion recognition, do not show better results for vad-emotion embedding. the reason from the primary observation is that it is due to overlaps in clusters, as one lexicon may belong to one or more emotions. also, at step-2, we retrofit word embeddings, only for emotion words present in nrc-emolex. the limitation with postprocessing methods such as counter-fitting is that it retrofits emotion words present in the constraints. hence, we should perform a global specialization or post-specialization processing @xcite for retrofitting non-emotion words with reference to the newly retrofitted emotion word vectors. this will turn into the retrofitting of complete vector space with the plutchik's emotion information, which may help to improve accuracy of the downstream tasks. at step-1 every word embedding is appended with vad-values or the neutral value vector [0.5,0.5,0.5] which retrofits every word for vadvalues in v' vector space. due to the limitation mentioned above, step-2 does not perform better than step-1. hence, the words which are not in the nrc-lexicons can not be retrofitted at step-2. to achieve the better accuracy results with final retrofitted vector space v", this is going to be our future work with the proposed method.