Disambiguating Emotional Connotations of Words Using Contextualized Word Representations (2024)

Abstract: Understanding emotional nuances in written content is crucial for effective communication; however, the context-dependent nature of language poses challenges in precisely discerning emotions in text. This study contributes to the understanding of how the emotional connotations of a word are influenced by the sentence context in which it appears. Leveraging the contextual understanding embedded in contextualized word representations, we conduct an empirical investigation to (i) evaluate the varying abilities of these representations in distinguishing the diverse emotional connotations evoked by the same word across different contexts, (ii) explore potential biases in these representations toward specific emotions of a word, and (iii) assess the capability of these representations in estimating the number of emotional connotations evoked by a word in diverse contexts. Our experiments, utilizing four popular models—BERT, RoBERTa, XLNet, and GPT-2—and drawing on the GoEmotions and SemEval 2018 datasets, demonstrate that these models effectively discern emotional connotations of words. RoBERTa, in particular, shows superior performance and greater resilience against biases. Our further analysis reveals that disambiguating the emotional connotations of words significantly enhances emotion identification at the sentence level.

## introduction understanding the emotional nuances conveyed by words is crucial for effective communication. this insight enhances the design of conversational agents that emulate human empathy, enabling responses that accurately reflect the emotions conveyed by word choice @xcite . psycholinguistics leverages this understanding to identify depression and suicidality, where specific word usage on social media posts may indicate underlying distress @xcite . moreover, comprehending these emotional subtleties alongside literal meanings of words can deepen second language comprehension for non-native speakers @xcite . in cognitive linguistics, the concept of emotional connotation refers to the emotion attributed to a specific word, transcending its explicit meaning @xcite . take, for instance, the word 'damn', which is rated by humans as anger @xcite , likely stemming from its frequent use in expressions of anger. however, a word may take on various emotional connotations depending on the context in which it appears. consider the word 'damn' in the following sentences sourced from the goemotions dataset @xcite s4. damn, that's dark here! @xcite in s1, the word 'damn' expresses anger, while in s2, it is used in a positive context to convey joy. both s3 and s4 exemplify its usage in other negative contexts. research on determining the emotional connotations of lexical items has typically utilized crowdsourcing methods, leading to the development of diverse lexicons of words with predefined emotions @xcite . these lexicons, however, provide static and generalized ratings for words, regardless of the context in which they are used @xcite . additionally, despite attempts to ensure consistency in word ratings through anchoring, implicit biases may persist in the rating process @xcite . efforts to address these limitations have mainly focused on distinguishing the polarity of words @xcite . in a domain-specific corpus (soccer), @xcite relied on human judgments to measure the differences between the positivity and negativ-ity of words with and without a sentence context. moreover, adopting automatic methods as an alternative to manual acquisition has been limited to extending the lexicon's word coverage @xcite or developing domain-specific polarity lexicons @xcite . recently, contextualized word representations, exemplified by bert @xcite , have been frequently evaluated on word relatedness benchmarks, such as word sense disambiguation @xcite , which is the task of identifying the correct sense of a word's usage from a fixed inventory of sense identifiers @xcite . studies on textual emotion analysis have utilized these representations particularly for sentence-level emotion classification tasks @xcite @xcite . what is particularly intriguing about contextualized word representations is their ability to generate unique embeddings for a word based on its context @xcite . our objective is to leverage this property of contextualized word representations to disentangle the emotional connotations of words that evoke various emotions within different sentence contexts. let w be the set of all words, where @xmath0, which evokes different emotions depending on its surrounding context. s is the set of all possible sentences, and n is the set of natural numbers representing the position in a sentence where we aim to analyze the emotional connotation of the word w. the function f , f (s, i) = e, categorizes the dominant emotional connotation of w at position i in sentence s. the signature of this function is given by f : s × @xmath1, where e is the set of all possible emotion categories. we conduct an empirical investigation to: (i) evaluate the varying abilities of contextual word representations in distinguishing the diverse emotional connotations evoked by the same word across different contexts; (ii) explore the existence of potential biases in these representations toward specific emotional connotations of a word; in this context, bias refers to the likelihood of models incorrectly associating a word linked to emotion k with emotion j ; (iii) assess the capability of these representations in estimating the number of emotions a word can evoke in various contexts; and (iv) investigate the impact of disambiguating the emotional connotations of words on the accuracy of sentence-level emotion classification. focusing on emotional words that evoke various emotions across diverse contexts, we obtain contextualized representations of these words within emotion-annotated sentences in the goemotions and the semeval 2018 @xcite datasets. we then cluster these representations and assess the alignment degree between the resulting clusters and the emotions of the words in question. our analysis of various models-bert, roberta, xlnet, and gpt-2-showcases their capability to capture the emotional connotations of words. we find that not all models are equally effective in discerning these emotional nuances. our findings also reveal biases towards specific emotions in these representations, with different models exhibiting biases towards different emotions for a given word. moreover, our experiments indicate that disambiguating the emotional connotations of a small number of words significantly improves the accuracy of sentence-level emotion classification. ## related work textual emotion recognition has typically involved either the utilization of lexicons-lists of words with pre-assigned emotions-without the need for extensive labeled data @xcite , or contextualized word representations, known for their domain-agnostic adaptability, when sufficient labeled data is available @xcite . methods that rely on lexicons view texts as word collections and use word ratings from lexicons for emotion identification @xcite . however, the static nature of these word ratings limits a comprehensive understanding of emotions, as they disregard contextual nuances @xcite . for instance, in a domain-specific corpus (soccer), @xcite demonstrated that pragmatic and semantic shifts in context can significantly influence word polarity in lexicons. to address this limitation, researchers often explore the identification of negations, diminishers, and intensifiers @xcite , or they develop domain-specific lexicons @xcite , which have mainly focused on distinguishing polarity of words in a specific domain @xcite . recent methods in textual emotion analysis have increasingly leveraged contextual word representations like bert, particularly for sentence-level emotion classification @xcite @xcite . these methods enhance model training by fine-tuning these embeddings with emotion-labeled datasets. for instance, @xcite applied these representations in a convolutional neural network to discern semantic relationships between words, and kassner and schütze (2020) focused on refining the understanding of contradictory sentiment words within these representations for binary sentiment classification. some studies have integrated emotional lexicons into these representations, enabling these models to achieve a more nuanced understanding of emotional words @xcite @xcite . for example, @xcite proposed a pretraining objective for bert, which increases masking probabilities for emotional words in sentences using emotion lexicons, while @xcite developed a bert model from scratch using yelp and amazon reviews by increasing the masking probability for positive and negative words. however, these approaches exhibit high sensitivity to both the training corpus and the lexical resources employed @xcite , which suffer from ambiguity @xcite . certain studies, such as @xcite , focused on semantic role labeling for emotions, modeling the semantics and interrelatedness of emotion labels by learning representations for emotion classes from annotated data. however, these representations do not generalize to other datasets and label formats @xcite . previous studies have primarily utilized contextualized representations for sentence-level emotion classification tasks. in contextualized word representations, each input word is represented as a vector dependent on the context of its occurrence @xcite . this approach captures both semantic and syntactic nuances within the surrounding context of words, rendering these models particularly intriguing for investigating the emotional connotations of words across diverse contexts. this paper exploits these representations to conduct an empirical study, aiming to scrutinize their efficacy in distinguishing different emotional connotations evoked by the same word in various contexts. to achieve this, we adopt a clustering-based approach, wherein the representation vectors of the word, obtained from emotion-annotated sentences, are clustered using a gaussian mixture model. further, we evaluate potential biases in different representa-tion models toward certain emotional connotations of words and assess whether clustering is a viable method for predicting the number of emotions a word can evoke in diverse contexts. ## methodology to investigate the effectiveness of contextualized word representations in discerning the various emotional connotations of words that evoke different emotions depending on the sentence context, we propose a method comprising the following steps: 1. target word identification: identify a subset of emotional words within an emotion lexicon that evoke diverse emotions across different contexts. 2. sample sentence extraction: retrieve sentences featuring the target words from emotion-annotated resources to compile a representative set of instances showcasing the words in diverse contexts. 3. contextualized representation generation: obtain contextualized representation vectors for the target words in the set of sample sentences. 4. word representation clustering. apply clustering to the contextualized representations using a gaussian mixture model (gmm) and find a mapping between the resulting clusters and the emotions of target words that maximizes the overall number of accurate matches, with the match rate serving as the evaluation metric. the next sections detail the target word identification phase, the word representations used in our study, and the clustering of these representations. ## experiments in this section, we first investigate the ability of various contextualized word representations to distinguish between the different emotional connotations evoked by the same word in different contexts (section 4.2). then, we explore the presence of emotional biases in these representations (section 4.3). finally, we evaluate the accuracy of these representations in quantifying the range of emotions elicited by each word (section 4.4). ## sentence-level emotion classification this section explores the impact of disambiguating the emotional connotation of words that evoke different emotions depending on the context, on the accuracy of sentence-level emotion detection. we evaluate sentences containing at least one identified target word, as those without these words remain unaffected. sentences are divided into stratified training (80%) and test (20%) splits based on emotions through random sampling. our initial experiments involve comparing the original nrc-affect lexicon and its modified versions in a before-and-after manner. here, modified lexicon refers to the disambiguation of emotional connotations associated with target words in the original nrc-affect lexicon, achieved by utilizing various contextualized word representations. the probability values from the gaussian mixture model indicate the extent to which each instance of a target word belongs to each of the gmm clusters, which have been mapped to specific emotions. we computed the average probability for all instances of a target word within an emotion's cluster to ascertain its disambiguated ratings. for example, while the original lexicon associated 'damn' exclusively with anger, with a score of 0.7, the modified lexicon provides a nuanced view of the different emotional connotations-joy, sadness, and fear, in addition to anger-that 'damn' evokes across various contexts in the goemotions dataset. building on the lexicon-based classifier design outlined in @xcite , we utilized the information from both the original nrc-affect lexicon and its modified versions as features in a logistic regression classifier for emotion prediction, detailed in appendix b. ## conclusion in this study, we have explored disentangling the emotional connotations of words within diverse sentence contexts, leveraging contextualized word representations. we evaluated these representations' ability to differentiate the diverse emotions of words, identify potential biases in predicting emotional connotations, and accurately estimate the multiplicity of words' emotional connotations. our methodology involved clustering based on contextualized representations of words that evoke different emotions in various contexts and assessing the alignment between the generated clusters and the words' emotions. our evaluation of bert, roberta, xlnet, and gpt-2 models revealed that contextualized representations can effectively disambiguate the emotional connotations of words, with roberta showing superior performance and greater resilience against biases. further analysis indicated that addressing a small subset of ambiguous emotional words and considering the context in determining their emotional connotations are crucial for accurately determining sentence emotion.