Cross-lingual Emotion Intensity Prediction (2020)

Abstract: Emotion intensity prediction determines the degree or intensity of an emotion that the author expresses in a text, extending previous categorical approaches to emotion detection. While most previous work on this topic has concentrated on English texts, other languages would also benefit from fine-grained emotion classification, preferably without having to recreate the amount of annotated data available in English in each new language. Consequently, we explore cross-lingual transfer approaches for fine-grained emotion detection in Spanish and Catalan tweets. To this end we annotate a test set of Spanish and Catalan tweets using Best-Worst scaling. We compare six cross-lingual approaches, e.g., machine translation and cross-lingual embeddings, which have varying requirements for parallel data – from millions of parallel sentences to completely unsupervised. The results show that on this data, methods with low parallel-data requirements perform surprisingly better than methods that use more parallel data, which we explain through an in-depth error analysis. We make the dataset and the code available at https://github.com/jerbarnes/fine-grained_cross-lingual_emotion .

## introduction emotion analysis within natural language processing attempts to identify the private states @xcite expressed in written text, which in many cases are only implicitly available. research often classifies these emotions into discrete categories @xcite , such as anger, fear, joy, or sadness. this discrete approach to emotion has been applied to fairy tales @xcite , headlines @xcite , and more recently micro-blogging services, such as twitter @xcite . however, people can express emotion in ways that require a more fine-grained approach than the basic discrete version. take the following two sentences: current state-of-the-art approaches to emotion intensity are based on supervised machine learning approaches, which combine several sources of annotated corpora, emotion and sentiment lexicons in order to achieve the best performance. however, the combination of all of these necessary resources is only available in a few high-resource languages, with english easily having the largest number. collecting a similar set of resources for all other languages is prohibitively expensive and would require years of work. therefore, it would be preferable to find a way to use the available resources in english to perform emotion intensity prediction in other languages. cross-lingual methods -either translation or cross-lingual embedding approaches -offer a possible solution to the lack of labeled data and have shown promise for sentiment analysis at document-level @xcite , sentence-level @xcite , and fine-grained @xcite . however, the greater number of classes in emotion classification and the difficulty of the regression task means that it is not obvious that the cross-lingual approaches that work well for sentiment analysis will necessarily work for cross-lingual emotion intensity prediction. in this work, we provide the first attempt at cross-lingual emotion intensity prediction, by comparing methods which rely on cross-lingual embeddings, machine translation, and unsupervised machine translation to transfer resources from english to predict the emotion in languages that do not have large available datasets or lexicon resources. for testing, we additionally annotate a dataset of tweets in spanish and catalan. our results show that surprisingly unsupervised machine translation is able to outperform both supervised machine translation and cross-lingual embedding methods on these datasets. we additionally perform detailed quantitative and qualitative error analyses of the supervised and unsupervised translation approaches, concluding that while the overall translation quality of the supervised system is better than the unsupervised system, it often does not translate hashtags, which are an important source of emotion information for this task. in the rest of the paper we discuss related work (section 2) and provide a description of the datasets (section 3) and models (section 4) used for the experiments. we then discuss the results (section 5) and provide an in-depth analysis of why certain models perform better (section 6). ## related work emotion detection attempts to identify explicitly or implicitly mentioned emotions in a text, either by following a proposed set of basic emotion categories @xcite @xcite or through valence-arousal approaches @xcite . however, in contrast to other tasks which also attempt to detect evaluative language, such as subjectivity or sentiment analysis, there are relatively few annotated resources, and most of these resources are found only in english @xcite @xcite . a notable exception is the deisear dataset @xcite , which crowdsources descriptions of emotional events in german. annotating categorical emotion is a subjective and complicated task, which often leads to low interannotator agreement @xcite . however, best-worst scaling has shown to improve overall agreement scores when annotating tweets @xcite . in this approach, annotators are shown n items (@xmath0, normally 4) and they must choose only the items that are best and worst, i. e. those that most and least represent the phenomena under question. despite these advances in annotating, for most languages in the world, there exists no annotated emotion dataset which could enable supervised emotion classification. on english data, previous approaches to classifying emotion have used word and character n-gram features @xcite , sentiment and emotion lexicon features @xcite , as well as a variety of neural networks @xcite @xcite . typically, strong emotion classification systems use a combination of these features to get the strongest performance. ## datasets for training the emotion intensity classifiers, we use the english data from the wassa 2017 shared task on emotion intensity prediction @xcite . the authors collected tweets and used crowd-annotation to achieve real-valued labels for four emotions (anger, fear, joy, and sadness). we use their predefined splits (statistics are shown in ## experimental setup in order to determine how much bilingual signal is required to predict crosslingual emotion intensity, we compare four methods with differing data needs. in the following, we describe these methods from those that require the largest amount of bilingual signal (mt) to those that require the least (unsup). supervised machine translation (mt): we use googletranslate foot_1 , which makes use of large amounts of parallel data, to translate the test samples to english. we then train a support vector regression model foot_2 on bag of words representations (mt-bow) and a second model with a number of additional features (mt-full). for the mt-full model, we include n-gram features (1-4), character n-grams (3-5), embedding features created by averaging the embeddings of all the tokens in the tweet, and finally features from the following lexicons: nrc hashtag sentiment lexicon @xcite , nrc hashtag emotion association lexicon @xcite and the nrc word-emotion association lexicon @xcite , where each feature is a real valued number which represents how much each word is associated to a polarity or emotion. the final representation of each tweet using the mt-full method is therefore a 65860 dimensional vector. finally, we train the svr model using the following settings (linear kernel, @xmath1. ## results the pearson correlation results are summarized for all models in the approach that obtains the highest overall pearson correlation across all emotions on both languages is unsup-full, averaging 0.37 on catalan and 0.32 on spanish. in addition, it is the best performing model on 4 of the 8 tasks, except for catalan joy, where mt-full is 0.01 percentage points (pp.) better, reaching 0.46 and catalan sadness (xlm-roberta is 0.04 pp. better, at 0.25) and catalan fear (xlm-roberta is 0.02 pp. better, at 0.42) and spanish sadness (blse is 0.02 pp. better, reaching 0.12). xlm-roberta is the second best model, averaging 0.35 and 0.30 on catalan and spanish, respectively, while mt-full is slightly worse (0.34 and 0.26). mt-bow and unsup-bow perform much worse (0.15/0.07 and 0.10/0.05), and the cross-lingual embedding methods are the worst by far (0.04/0.03 for cwe and 0.07/0.15 for blse). it is clear that the additional features (character n-grams, embedding features, and lexicon features) are essential. mt-full performs an average of 0.19 pp. pearson better than mt-bow, while unsup-full leads to 0.28 pp. improvement over unsup-bow. we further confirm this in section 6.2. regarding the cross-lingual embedding models, it seems evident that these do not contain enough information to accurately predict emotion intensity in the target language. blse does outperform cwe on both catalan and spanish (an average of 0.03 pp. and 0.12 pp., respectively) and both mt-bow and unsup-bow on catalan (0.08 pp. and 0.10 pp.), but the overall performance is still poor. these models are also the poorest performers monolingually. there is large performance gap between xlm-roberta and mbert, on both the monolingual (0.27 pp.) and cross-lingual tasks (0.38/0.22 pp.), as mbert is the weakest cross-lingual model and xlm-roberta the second best. additionally, there is a divergence between sadness and the rest of the emotions analyzed, with no model achieving more than 0.21 or 0.12 in catalan and spanish. this seems to indicate that sadness may be harder to classify cross-lingually, as monolingually this class is has the best classification results @xcite . this class also has the fewest training and development examples in english, which may indicate that the good previous results monolingually may have been due to overfitting to the data. it is also possible that the particulars of the target language test data are the reason for this difference, although the inter-annotator agreement scores suggest that sadness is not more difficult than the other classes. ## analysis in this section we compare both quantitatively and qualitatively the differences in translation quality between mt and unsup. furthermore, we perform an ablation study to determine which features are the most important for mono, mt-full, and unsup-full. ## conclusion in this paper, we provided the first attempt at cross-lingual emotion intensity prediction, by comparing methods which rely on differing amounts of cross-lingual signal, ranging from millions of parallel sentences (mt), small bilingual dictionaries (cross-lingual embeddings), to no explicit cross-lingual signal at all (unsup). we compare these methods on two target languages, spanish and catalan, which do not have large available emotion datasets or lexicon resources. in order to test the models, we additionally annotated a small dataset of tweets in spanish and catalan. our results show that translation methods outperform embedding-based methods for almost all emotions and achieve reasonable average results, although there is still a noticeable gap to reach monolingual levels. surprisingly, unsupervised translation is the best performing cross-lingual method, largely due to the fact that it more accurately translates hashtags. xlm-roberta performs nearly as well, but unfortunately cannot be combined with sentiment and emotion lexicons available in english, as it processes the original target-language data. these results may not hold for other domains, such as literature or opinion pieces, where emotional information is not concentrated in a similar way. in the future, it would be interesting to perform experiments on various domains, in order to determine whether unsupervised machine translation for cross-lingual emotion is robust to domain shift. theoretically, this is simpler for unsupervised mt rather than supervised mt, which could motivate further research in this direction. as lexicon information has proven so useful for this task, it could be interesting to look into approaches that use this information to improve pretrained multilingual language models. additionally, given the importance of hashtags for emotion detection in tweets, it would be important for future work in cross-lingual emotion detection to concentrate on achieving better translations of hashtags. finally, we contemplate promising research on emotion detection and classification using the newly annotated data in catalan and spanish we introduce here. we expect that this will contribute to furthering research on these two languages.