Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis (2023)

Abstract: Emotion arcs capture how an individual (or a population) feels over time. They are widely used in industry and research; however, there is little work on evaluating the automatically generated arcs. This is because of the difficulty of establishing the true (gold) emotion arc. Our work, for the first time, systematically and quantitatively evaluates automatically generated emotion arcs. We also compare two common ways of generating emotion arcs: Machine-Learning (ML) models and Lexicon-Only (LexO) methods. By running experiments on 18 diverse datasets in 9 languages, we show that despite being markedly poor at instance level emotion classification, LexO methods are highly accurate at generating emotion arcs when aggregating information from hundreds of instances. We also show, through experiments on six indigenous African languages, as well as Arabic, and Spanish, that automatic translations of English emotion lexicons can be used to generate high-quality emotion arcs in less-resource languages. This opens up avenues for work on emotions in languages from around the world; which is crucial for commerce, public policy, and health research in service of speakers often left behind. Code and resources: https://github.com/dteodore/EmotionArcs

## introduction commercial applications as well as research projects often benefit from accurately tracking the emotions associated with an entity over time. public health researchers are interested in analyzing social media posts to better understand populationlevel well-being (vishnubhotla and mohammad, 2022), loneliness (guntuku et al., 2019), depression (de choudhury et al., 2013), etc. government policy makers benefit from tracking public opinion over time for developing effective interventions and laws. for example, tracking sentiment towards health interventions such as mask mandates and vaccine policies (hu et al., 2021). researchers in digital humanities are interested in understanding basic components of stories such as plot structures, how emotions are associated with compelling characters, categorizing stories based on emotion changes (reagan et al., 2016), etc. in all of these applications, the goal is to determine whether the degree of a chosen emotion has remained steady, increased, or decreased from one time step to the next. the time steps of consideration may be days, weeks, years, etc. this series of time step-emotion value pairs, which can be represented as a timeseries graph, is often referred to as an emotion arc (mohammad, 2011; reagan et al., 2016). automatic methods generate emotion arcs using: the arcs are generated through a two-step process: a robust and simple setup for evaluating emotion arcs also allows one to test different design decisions in how the arcs are generated. for example, we know little about how best to aggregate information when using emotion lexicons; e.g., is it better to use coarse or fine-grained lexicons, should we ignore slightly emotional words, etc. work on emotions has largely focused on english, excluding the voices, cultures, and perspectives of a majority of people from around the world. such an exclusion is not only detrimental to those that are excluded, but also to the world as a whole. for example, western-centric science that only draws on data from western languages may falsely claim certain universals about language, emotions, and stories, that when tested on data from other cultures does not hold true anymore. thus, working with more languages helps discover new paradigms that represent the world better. however, generating emotion arcs in most languages other than english is stymied by the lack of emotion-labeled resources. using translated labeled texts with ml methods is problematic (mohammad et al., 2016; hogenboom et al., 2014). however, for a number of nlp tasks, systems have benefited from using translations of lexical resources (from say english) into the target language. thus, high-accuracy lexo methods for creating emotion arcs can open up avenues of work in various languages using (manual or automatic) translations of existing emotion lexicons. our work, for the first time, systematically and quantitatively evaluates automatically generated emotion arcs. we also compare the ml and lexo methods for generating arcs. note, that our aim is not to build more accurate emotion classification systems, but rather we apply existing methods to evaluate emotion arcs generated across domains and languages. we conducted experiments on 18 datasets in nine languages from diverse domains, including: tweets, sms messages, reviews, etc. (two english datasets from movie reviews, four english datasets from semeval 2014, six datasets from semeval 2018 (two each in english, arabic, and spanish), and six african languages datasets from semeval 2023 (afrisenti).) emotions can be characterized by various dimensions and categories (e.g., valence, arousal, dominance vs. anger, fear, joy, sadness). in this work we explore the valence (or sentiment), and therefore generate valence (or sentiment) arcs. @xcite our work sets the foundation for future work which can extend analyses to various emotion categories. ## related work emotion arcs have commonly been created from literary works and social media content. alm and sproat (2005) were the first to automatically classify sentences from literary works for emotions using a machine learning paradigm. mohammad (2011) was the first to create emotion arcs and analyse the flow of emotions across the narrative in various novels and books using emotion lexicons. kim et al. (2017) built on this work by creating emotion arcs to determine emotion information for various genres using the nrc emotion lexicon (mohammad and turney, 2013). @xcite reagan et al. (2016) and del vecchio et al. (2018) clustered emotion arcs and found evidence for six prototypical arc shapes in stories. hipson and mohammad (2021) analysed emotion arcs for individual characters (instead of the whole narrative) in movie dialogues. emotion arcs of literary works have been used to better understand plot and character development (reagan et al., 2016; kim et al., 2017; hipson and mohammad, 2021); and also for assisting writers develop and improve stories (ashida et al., 2021; somasundaran et al., 2020). recently, hipson and mohammad (2021) introduced how the patterns with which emotions change over time -emotion dynamics -can be inferred from text. vishnubhotla and mohammad (2022) computed ueds from tweets and analysed how they have changed over the years. teodorescu et al. (2023b) studied emotional development in children's writing and found meaningful patterns across age. in the public health domain, teodorescu et al. (2023a) created emotion arcs for tweeters who self-disclosed as having a mental health diagnosis (on twitter). they found that certain emotion dynamics patterns differed significantly between tweeters with mental health conditions (e.g., attention deficit hyperactivity disorder, depression, bipolar) compared to the control group. however, there is surprisingly little work on arc evaluation. a key reason for this is that it is hard to determine the true emotion arc of a story from data annotation. one attempt to evaluate aspects of an emotion arc can be seen in bhyravajjula et al. (2022). they asked one volunteer to read minisegments of a 'the lord of the rings' novel to determine whether the protagonist's circumstance undergoes a positive or negative shift. they then determined the extent to which the automatic method captured the same shifts. here we propose a simpler alternative way to robustly evaluate automatically generated emotion arcs on a wide variety of domains and multiple languages. ## experimental setup to evaluate automatically generated emotion arcs we begin by describing the evaluation setup. this is a key contribution of this work since no prior work systematically evaluates automatically generated emotion arcs. we construct gold emotion arcs from existing datasets where individual instances are manually annotated for valence (sentiment). here, an instance could be a tweet, a sentence from a customer review, a sentence from a personal blog, etc. depending on the dataset, manual annotations for an instance may represent the emotion of a speaker, or sentiment towards a product or entity. for a pre-chosen bin size of say 100 instances per bin, we compute the gold emotion score by taking the average of the human-labeled emotion scores of the instances in that bin (in-line with the commerce and social media use cases discussed earlier). we move the window forward by one instance, compute the average in that bin, and so on. (using larger window sizes does not impact conclusions.) we created text streams for the experiments in section 4 by ordering instances from a dataset by increasing gold score before binning. this created arcs with a more consistent emotion change rate and we first explore results in this scenario. in section 7, we look at the impact of more dynamic emotion changes (e.g., varying peak heights and widths) on performance. we automatically generate emotion arcs using the lexo and ml methods discussed in the two sections ahead. we standardized all arcs (aka zscore normalization) so that the gold arcs are comparable to automatically generated arcs. @xcite finally, we evaluate the closeness of automatically generated emotion arcs with gold arcs using two metrics: linear correlation and root mean squared error (rmse). we use spearman rank correlation (spearman, 1987) (range: -1 to 1). high correlation implies greater fidelity: no matter what the overall shape of the gold emotion arc, when it goes up (or down), the predicted emotion arc also goes up (or down). foot_3 rmse (range: 0-∞) is a measure of the error between the true and predicted values. it penalizes bigger errors more, and thus is sensitive to outliers. scores closer to 0 indicate better predictions. in our experiments, the correlation and rmse scores are determined for thousands of points between the predicted and gold arcs. @xcite in all, we conducted experiments with 18 emotion-labeled datasets from nine languages, with labels that are either categorical or continuous, for valence. the results also establish key benchmarks; useful to practitioners for estimating the quality of the arcs under various settings. ## lexo arcs: emotion arcs generated from counting emotion words recall that in the introduction we described how emotion arcs are automatically generated. key parameters in that process include bin size, type of emotion lexicon used (e.g., categorical or continuous emotion scores), and how to handle terms in the text that are not in the lexicon (oov terms). choice of bin size depends on the application and available data. for example, if a company wants to know how the proportion of angry comments has changed from month to month in the last 10 years, then the bin size to use is month. if instead, they want to know how things changed on a day-by-day basis over the last 45 days, then the bin size to use is a day. depending on the volume of relevant data and the bin size chosen, the bins may include a higher or lower number of instances. the two oov handling methods explored were: 1. assign label na (no score available) and disregard these words, and 2. assign 0 score (neutral or not associated with emotion category), thereby, leading to a lower average score for the instance than if the word was disregarded completely. we then evaluated how closely the arcs correspond to the gold valence arcs. in appendix b, we show the impact of using emotion lexicons more selectively-using only the entries that have an emotion association greater than some predetermined threshold. results (valence): figure 1 shows the correlations and rmse values between predicted valence arcs and the gold valence arcs. bin size: overall, across datasets and regardless of the type of lexicon used and how oov words are handled, increasing the bin size dramatically improves correlation with the gold arcs. in fact, with bin sizes as small as 50, many of the generated arcs have correlations above 0.9. with bin size 100 and above correlations approach high 0.90's. similarly, rmse starts at approximately 1.20 to 1.10 at bin size 1, and quickly drops as bin size increases. at bin size 300, rmse approaches quite low scores (0.3-0.1). if we use a bin size of 1, we only get the ups and downs of the emotions associated with individual words, which will likely be quite far off from the true emotion arc. as bin size increases (more data per bin), the predicted arc gets closer to the true arc probably because: if bin x is expressing more of an emotion (higher emotion intensity) than bin y, then x has a higher average word-emotion association score than y. categorical vs. real-valued lexicons: using a realvalued lexicon obtains higher correlations across bin sizes, methods for processing oov terms, and datasets. the difference is marked for very small bin sizes (such as 1 and 10) but progressively smaller for higher bin sizes. entries from realvalued lexicons carry more fine-grained emotion information, and it is likely that this extra information is especially helpful when arcs are determined from very little text (as in the case of small bins). oov terms: results with the two oov-handling methods were comparable (no clear winner). discussion: for many social media applications, one has access to tens of thousands, if not millions of posts. there, it is not uncommon to have timesteps (bins) that include thousands of instances. thus, it is remarkable that even with relatively small bin sizes of a few hundred, the simple lexicon approach is able to obtain very high correlations. of course, the point is not that the lexicon approach is somehow special, but rather that aggregation of information can very quickly generate high quality arcs, even if the constituent individual emotion signals are somewhat weak. ## ml arcs: emotion arcs generated from counting ml-labeled sentences this section explores how the accuracy of instancelevel (sentence-or tweet-level) emotion labeling impacts the quality of the generated emotion arcs. we approached this by creating an 'oracle' system, which has access to the gold instance emotion labels. there are several metrics for evaluating sentiment analysis at the instance level such as accuracy, correlation, or f-score. however, we focus on accuracy as it is a simple intuitive metric. inputs to the oracle system are a dataset of text (for emotion labelling) and a level of accuracy (e.g., 90% accuracy) to perform instance-level emotion labelling at. then, the system goes through each instance and predicts the correct emotion label with a probability corresponding to the pre-chosen accuracy. in the case where the system decides to assign an incorrect label, it chooses one of the possible incorrect labels at random. we use the oracle to generate emotion labels pertaining to various levels of accuracy, for the same dataset. we then generate emotion arcs just as described in the previous section (by taking the average of the scores for each instance in a bin), and evaluate the generated arcs just as before (by determining correlation with the gold arc). this oracle system allows us to see how accurate an instance level emotion labelling approach needs to be to obtain various levels of quality when generating emotion arcs. figure 2 shows the correlations of the valence arcs generated using the oracle system with the gold valence arcs created from the semeval 2018 v-oc test set (that has 7 possible labels: -3 to 3). @xcite we observe that, as expected the oracle systems with instance-level accuracy greater than approximately the random baseline (14.3% for this dataset) obtain positive correlations; whereas those with less than 14% accuracy obtain negative correlations. as seen with the results of the previous section, correlations increase markedly with increase in bin size. even with an instance-level accuracy of 60%, correlation approaches 1 at larger bin sizes. overall, we again observe high quality emotion arcs with bin sizes of just a few hundred instances. at bin sizes 200 and 300. however, for a given application scenario, these results can only be obtained when the machine learning system is able to train on sufficient domain-specific training data (which is often scarce), the computer power, and knowledge to work with these systems is accessible. for applications where simple, interpretable, low-cost, and low-carbon-footprint systems are desired, the lexicon-based systems described in the previous section, are often more suitable. ## lexo arcs: emotion arcs generated from translated emotion lexicons given the competitive performance of the lexicon-only (lexo) method, and the many benefits of the lexo method such as their simplicity and interpretability, we now explore whether high-quality emotion arcs can be created for low-resourced languages using automatic translations of english emotion lexicons. specifically, we make use of translations of the nrc lexicons from english into the language of interest and perform similar experiments as described in the previous section. @xcite we explore bin sizes of 400 and 500 as well, as we expect that instance-level accuracy will be lower in the translated-lexicon case. we would like to note that automatic translations may not be available for many very low resource languages. currently, google translate has functionality to translate across about 120 languages. this may seem large (and it is indeed a massive improvement from just a decade back), but thousands of languages still remain on the other side of the digital divide. thus, even requiring word translations is a significant limitation for many languages. we generated and evaluated emotion arcs in six african languages: amharic, hausa, igbo, kinyarwanda, swahili, and yoruba (using the datasets shown in ## arc quality under varying emotion amplitudes and spikiness aspects of how emotions change across the bins also impacts the quality of generated arcs. for example, for a given surge or dip of emotion, if the change of emotion strength is too small in magnitude (small amplitude) or occurs for too short of a time period (high spikiness), then the automatic method may fail to register the surge/dip. we will refer to arcs with many such emotion changes as more dynamic than those with more gradual and longer-duration emotion changes. recall that in section 4 we used text streams ordered by increasing gold score. now, to test the robustness of the lexo method, when dealing with dynamic emotion changes, we created new text streams by reordering the tweets in the emotionlabeled datasets in more random and dramatic ways. we created these dynamic text streams by sampling tweets from the chosen dataset with replacement until 200 crests and 200 troughs of various amplitudes and spikiness were obtained. @xcite the gold arc is then standardized as before. we will refer to this new text stream generated from a dataset as [dataset_name]-dynamic, and the gold arcs created with this process as dynamic gold arcs. the gold line in figure 4 shows the beginning portion of the gold valence arc produced from the semeval 2018 (v-reg) dataset (bin @xmath0. observe that some peaks are rather small in amplitude whereas some others are greater than two standard deviations. also, some emotion changes occur across a wider span of data (x-axis) whereas some others show large emotion change and back in a short span. we then generated the predicted arc for this dataset using the lexo method, the nrc vad lexicon, and by ignoring terms not found in the lexicon when determining bin scores. the green line in figure 4 shows the beginning portion of the gold valence arc when using a rolling window with bin size 100. observe that the green line traces the gold line quite closely. there are also occasions where it overshoots or fails to reach the gold arc. figure 4 shows gold and predicted arcs for the hausa dataset (bin @xmath1. observe that here the predicted arc still follows the gold arc closely (but as expected, not as closely as a.) in experiments modifying bin size, the predicted arcs obtain a high correlation (above 0.9) with a bin size of 100 (and above), even for the dynamic text stream. however, these scores are (as expected) consistently lower than what were observed for the corresponding non-dynamic text stream. note that the difference in scores is more pronounced for the smaller bin sizes than for larger ones. therefore, in scenarios with smaller amounts of data per bin, judicious use of parameters described in section 4 (e.g., treating oovs as neutrals, using real-valued lexicons, etc.) can have marked influence on arc quality. overall, though, these results show that the lexo method performs rather well even in the case of highly dynamic emotion changes. ## concluding remarks this work made contributions in two broad directions: first, we showed how methods for predicting emotion arcs can be evaluated, and used it to systematically and quantitatively evaluate both lexicon-only and machine learning methods. second, we showed that using translations of english lexicons into the language of interest can generate high quality emotion arcs, even for languages that are very different from english. both ml and lexo methods produced highquality emotion arcs when using bin size 50 and above (e.g., with bin size 100: obtaining correlations with gold valence arcs exceeding 0.98). ml methods are able to obtain markedly higher correlations at very low bin sizes (<50). however, with the abundance of textual data available from social media, for many applications where one is aggregating hundreds or thousands of instances in each bin, the gains of ml methods in terms of correlation scores are miniscule. with the cross-lingual experiments, we created emotion arcs from texts in six indigenous african languages, as well as arabic and spanish. for all of them, emotion arcs obtained high correlations as the bin size (aggregation level) was set to at least a few hundred instances. correlations between predicted and gold emotion arcs were in general higher when the target language was closer to english. in the last part of the paper, we explored how depending on the data available and the desired granularity of emotion arcs interacts with performance for various arc amplitudes and arc spikiness. we showed that the lexo method performs rather well even in the case of highly dynamic emotion changes. however, for smaller bin sizes, the predicted arc may markedly overshoot or not fully capture the gold peak or trough. in all, we conducted experiments with 18 sentiment-labeled datasets from 9 languages, thus the conclusions drawn are rather robust. the results on individual datasets also establish key benchmarks; useful to practitioners for estimating the quality of the arcs under various settings. finally, since this work shows that simple lexicon-only approaches produce accurate arcs, practitioners from all fields, those working in low-resource languages, those interested in interpretability, or those without the resources to deploy neural models, can easily generate high-quality emotion arcs for their data.