Misery Loves Complexity: Exploring Linguistic Complexity in the Context of Emotion Detection (2023)

Abstract: Given the omnipresence of social media in our society, thoughts and opinions are being shared online in an unprecedented manner. This means that both positive and negative emotions can be equally and freely expressed. However, the negativity bias posits that human beings are inherently drawn to and more moved by negativity and, as a consequence, negative emotions get more traffic. Correspondingly, when writing about emotions this negativity bias could lead to expressions of negative emotions that are linguistically more complex. In this paper, we attempt to use readability and linguistic complexity metrics to better understand the manifestation of emotions on social media platforms like Reddit based on the widely-used GoEmotions dataset. We demonstrate that according to most metrics, negative emotions indeed tend to generate more complex text than positive emotions. In addition, we examine whether a higher complexity hampers the automatic identification of emotions. To answer this question, we fine-tuned three state-of-the-art transformers (BERT, RoBERTa, and SpanBERT) on the same emotion detection dataset. We demonstrate that these models often fail to predict emotions for the more complex texts. More advanced LLMs like RoBERTa and SpanBERT also fail to improve by significant margins on complex samples. This calls for a more nuanced interpretation of the emotion detection performance of transformer models. We make the automatically annotated data available for further research at: https://huggingface.co/datasets/pranaydeeps/CAMEO

## introduction the negativity bias, coined by @xcite , states that human beings are inherently drawn to, and impacted more by negative experiences, emotions, and interactions than by their positive counterparts. this cognitive bias may be more present than ever because of the evolution of social media which makes sharing thoughts and opinions easier than ever before. recent research by @xcite has shown that publishing negative criticism or insults against political opposition is a highly effective way to acquire engagement on social media platforms. similar research in the news domain shows that headlines with negative keywords generate more engagement and clicks than headlines with positive keywords @xcite . while these studies explore the negativity bias from a consumer's perspective, in this work we explore the idea of a negativity bias from the perspective of the author and more specifically explore the correlations between negativity and complex language use. the goal of this research is thus to further investigate the negativity bias in social media texts by exploring whether there is a link between emotion and text complexity. this overarching research goal will be broken down into two specific research questions. (1) are negative emotions on social media more complex than positive emotions? (2) does a text's linguistic complexity make it harder to model certain emotions using transformers, and do transformers that are better at understanding more complex text -such as spanbert @xcite -perform better than bert @xcite on linguistically complex emotional text? firstly, we attempt to understand, in a strictly linguistic sense, the differences in the ways negative and positive emotions are manifested on social media. to investigate this we rely on the goemotions dataset @xcite , which comprises english reddit comments that have been manually annotated with 28 emotion categories. we investigate these texts' linguistic complexity by relying on diverse metrics from readability research. more specifically we consider a text more complex according to both lexical and syntactic complexity metrics. in this first stage of our research, we investigate possible correlations between these metrics and the polarity of the gold-standard emotion labels. in the second stage, we fine-tune three state-of-the-art transformers (bert, roberta, and spanbert) on the emotion dataset to perform multi-label classification. we demonstrate that according to most metrics, negative emotions indeed tend to generate more complex text than positive emotions and that the transformer models are generally worse at predicting emotions for the more complex texts. we believe this calls for a more nuanced interpretation of the emotion detection performance of transformer models, with more detailed evaluations to have a better understanding of performance for complex data. the remainder of this paper is organized as follows. section 2 gives a brief overview of relevant related research, whereas section 3 describes the data and information sources used for the experiments and analyses. section 4 elaborates on the correlations between complexity and emotion polarity, whereas section 5 discusses the machine learning experiments carried out to evaluate the performance of transformers on instances with various degrees of complexity. section 6 ends this paper with concluding remarks and indications for future research. ## extracting complexity metrics to investigate the link between emotions and complexity, we extracted a set of linguistic text characteristics that have shown to be good predictors for complexity in readability research (see section 2.2). in the remainder of this paper, we will refer to these features as complexity metrics. in this section, we will first describe the datasets used for the correlation analysis and emotion detection experiments, and subsequently present an overview of all metrics that were extracted to measure complexity. ## correlation between linguistic complexity and negative emotions in this section, we investigate whether there is a positive correlation between linguistic complexity and the presence of negative emotions. as explained in section 3, we rely on syntactic and lexical readability metrics to measure complexity, while dividing the 28 emotion labels into 3 sets: positive (12 labels), negative (11 labels), and neutral and ambiguous (5 labels). as a preliminary attempt to study these correlations, we first perform an exploratory factor analysis (efa) to justify the use and selection of the complexity metrics before proceeding to investigate links between complexity and emotions. moreover, by classifying the metrics into factors, we can further analyze the difficulties of transformers in modelling certain types of complexity. first, we perform barlett's test of sphericity (armstrong and @xcite ) to confirm the validity of the factor analysis. the p-value was found to be 0.0 which means the observed correlation matrix is not an identity matrix and the test is statistically significant. we also perform a keyser-meyer-olkin (kmo) test which measures the proportion of variance among all the observed variables. the kmo for our data was found to be 0.659. a value higher than 0.6 is usually considered adequate for factor analysis. to determine the adequate number of factors for decomposition, we set up a scree plot, which is shown in figure 2 . since 3 factors seem to have an eigenvalue higher than 1.0, we perform the factor analysis with 3 components. figure 1 visualizes the factor loading matrix, demonstrating that factor 1 largely combines syntactic metrics, while factor 2 focuses on frequency-based metrics like tf-idf, log-likelihood, and type token ratio and factor 3 solely focuses on basic lexical metrics like word length and polysyllable occurrences. the factor variance reveals a cumulative value of 0.413, which means that a cumulative variance of 41.3% of the data is explained by the 3 factors. in section 5 we further use the distinction between the factors to analyze the difficulties of transformer models in modeling certain types of linguistic complexity. since the metrics used seem to adequately represent linguistic complexity as statistically proven by the efa, we can use these metrics to further understand the relationship between complexity and emotion polarities. to this end, we divide the complexity-enriched goemotions dataset into three polarities and calculate the average value for each metric for the three polarities individually. figure 3 displays the averages for all 12 metrics for the three polarities. according to the figure, 9 out of the 12 metrics indeed have a higher average score for negative polarity. for some metrics, like left-embededness, nested noun distance, or dale word frequency, the difference between positive polarity averages is strikingly lower than the negative and neutral polarity averages. while avg. tf-idf, avg. log-likelihood and type token ratio % are anomalies, all three of these metrics convey correlated information represented by factor 2 in the efa (as illustrated in figure 1 ). ## emotion detection evaluation in this section, we attempt to understand how different state-of-the-art transformers are affected by complexity when classifying emotions using the goemotions datasets, which constitutes with its 28 emotion categories a very fine-grained classification task. for these experiments, we investigate 3 major transformers, bert @xcite , spanbert @xcite , and roberta @xcite . we fine-tune each of these models for multi-label classification for 20 epochs, with an initial learning rate of 2e -5 and a weight decay of 0.1, on a train set of 46,407 samples (80%) from the raw version of the goemotions dataset. we then use the best-performing checkpoint on a validation set of about 5,800 samples (10%). as an evaluation set, we use the remaining 5,802 samples (10%). all scores reported are averaged over 3 runs with different seeds. to evaluate the output of the three fine-tuned transformer models, we again incorporate the metrics described in section 3.2. we divide metrics using the efa described in section 4 into the 3 factors, each representing a different aspect of linguistic complexity, i.e., syntactic complexity the detailed emotion classification results, which are summarized in a second noteworthy observation is that the factor 1 simple subsets tend to frequently outperform the factor 1 complex and medium subsets, as illustrated by figure 4(a) . this is again verified by a two-tailed sample t-test, which proves statistically significant at @xmath1.05. this is in accordance with the presumption that complex text samples require a higher degree of natural language understanding (nlu), which the current generation of transformers is not (yet) capable of. this observation seems contrasted by the better performance on most factor 2 complex subsets, which often even surpasses the results obtained on the factor 2 simple subsets. this, however, might be linked to the fact that lexical complexity also comes with lexical richness, i.e., the presence of distinguishing keywords seldom used, which was one of the complexity metrics (type-token ratio) we used to categorize the evaluation set according to complexity, and therefore potentially a key factor in the decision-making for ed. whereas, for factor 3, the difference in simple vs. complex subsets is inconclusive. this suggests that modeling lexically complex samples is not a pertinent issue of these sets of transformers. the final, rather disappointing, observation is that spanbert, which is designed to better represent and predict spans of text, and thus assumed to better model more complex syntax, does not obtain better classification scores than bert or roberta on the complex subsets. in fact, span-bert only outperforms bert and roberta for some emotionally neutral subsets, which might point to the pre-training data lacking emotional richness. spanbert was trained on the same datasets as bert, but the data was restricted to samples exceeding 512 sub-tokens only from book-corpus and wikipedia, potentially inhibiting the ability of the model to understand non-standard short-form social media communication. ## conclusion in this work, we attempted to answer two research questions on the relationship between linguistic complexity and emotion detection. for this purpose, we relied on the goemotions dataset, which is a high-quality benchmark dataset for emotion detection that has been manually annotated with 28 emotion categories. in the first part of this research, we investigated whether negative emotions are more linguistically complex than positive ones. with the help of twelve complexity metrics sub-divided into three factors, we discovered that negative emotions indeed have a higher average degree of complexity than positive or neutral emotions. secondly, we attempted to dive deeper into the evaluation of fine-tuned transformers for emotion detection, by dividing our evaluation set from the goemotions dataset into complex, medium, and simple samples. through this nuanced evaluation of three state-of-the-art transformers (bert, spanbert and roberta), we find that syntactically complex samples are often misclassified. we also notice that irrespective of difficulty, all models tend to perform poorly on negative emotions. finally, we also discover that even though spanbert encodes a higher degree of complexity, this does not extend to emotion detection of complex text, as it either performs worse or is comparable to transformers trained with standard masking (bert and roberta). to conclude, we aim to encourage a more detailed evaluation of current emotion detection systems, by extending the evaluation setup to better understand the performance for more linguistically complex instances, rather than the reporting of aggregated scores which may only paint a halfformed picture. in future work, we would like to demonstrate the validity of the results on datasets from other domains and annotation strategies, as well as languages other than english. further, since linguistic complexity does have an impact on emotion detection performance, we would like to investigate more intricate approaches to model more complex samples.