Universal Joy A Data Set and Results for Classifying Emotions Across Languages (2021)

Abstract: While emotions are universal aspects of human psychology, they are expressed differently across different languages and cultures. We introduce a new data set of over 530k anonymized public Facebook posts across 18 languages, labeled with five different emotions. Using multilingual BERT embeddings, we show that emotions can be reliably inferred both within and across languages. Zero-shot learning produces promising results for low-resource languages. Following established theories of basic emotions, we provide a detailed analysis of the possibilities and limits of cross-lingual emotion classification. We find that structural and typological similarity between languages facilitates cross-lingual learning, as well as linguistic diversity of training data. Our results suggest that there are commonalities underlying the expression of emotion in different languages. We publicly release the anonymized data for future research.

## introduction emotions are fundamental to human experience across languages and cultures. the nature of emotions and their linguistic expression is a topic of enduring interest across disciplines such as psychology, linguistics, philosophy, and neuroscience. emotion researchers have investigated the existence of basic emotions such as anger, fear, disgust, sadness, and happiness @xcite , all of which were already described in the 19th century by @xcite and @xcite . furthermore, @xcite reports a growing consensus concerning the universality of emotions across languages and cultures. computational linguistics can help shed light on the way in which emotions are expressed in the languages of the world. however, most existing research has focused either on english or used very small multilingual data sets. we present a new dataset, universal joy (uj), of over 530,000 anonymized public facebook posts distributed across 18 languages, labeled with five different emotions: anger, anticipation, fear, joy, and sadness. this dataset represents a substantial advance over prior datasets, both in terms of its size and its linguistic diversity. it provides a strong empirical foundation for exploring basic questions about the nature and expression of emotions across the languages of the world. figure 3 shows the heatmap of relative emotion distribution for each language. in this paper, we use this dataset to explore multilingual emotion classification. we first perform emotion classification in a monolingual setting, i.e., training and testing on a single language. we then expand to a cross-lingual setting, i.e., where the training data contains other languages in addition to the test data language. finally, we test how well we can do in a zero-shot learning setting; here the training data does not include any data in the language of the test data -a setting particularly relevant for low-resource languages. overall, we find consistent effects of crosslingual learning, which raises several interesting issues: first, it suggests that accurate emotion detection might be possible even for low-resource languages. accurate models for such languages might be achievable with substantial amounts of training data from high-resource languages like english. more generally, however, we explore why cross-lingual learning works, and what linguistic circumstances support or hamper such learning. we explore three main factors: code-switching, typological closeness of training and test languages, and linguistic diversity in the training data. we hope the richness of this data set opens up exciting future research avenues, and release the models and the complete anonymized dataset at https: //github.com/sotlampr/universal-joy. contributions 1) we publish a new dataset of over 530k anonymized public facebook posts in 18 languages, labeled with five basic emotions. 2) we show results for various classification setups, including transfer learning setups like cross-lingual and zero-shot learning. 3) we analyze the sources of cross-lingual learning in depth, including the effect of code-switching, typological closeness, and linguistic diversity. ## data the dataset described here is a substantially reorganized and cleaned version of one previously described, but not released @xcite . it was collected in october 2014 by searching for public facebook posts with a facebook "feelings tag". we did verify publication with the data protection officer of the main institution. for a data statement @xcite , see appendix a. we remove any duplicates, and classify each instance's language using three methods: langid, foot_0 cld3, foot_1 , and fasttext. 3 we keep only instances where at least two of these methods agree. we manually evaluate 200 randomly selected instances labeled deu, fra, eng, ita, and spa, foot_3 and find the average precision of our method is 0.97(±0.04). to anonymize the data, we remove identifying information by replacing names with the special token @xcite . where possible (dutch, english, french, german, portuguese, spanish, italian), we use the spacy foot_4 ner to replace any person entities. for languages without spacy support, we either use the stanford corenlp ner tagger @xcite and replace person-tagged words (chinese), or replace all given names and surnames found in wiktionary for the respective languages (bengali, burmese, hindi, indonesian, khmer, malay, romanian, tagalog, thai, vietnamese). finally, we perform some additional preprocessing steps to replace any number with 0, and the facebook-specific tags "with [person]" with the special token similar to the approach of @xcite , we map the 27 initial emotion tags into five labels of basic emotions: anger, anticipation, fear, joy, and sadness (see mapping in appendix as training languages, we choose all languages with more than 35 samples for the least frequent emotion (i.e., fear). the size and distribution of emotions for each dataset is presented in ## methods we model the task as a series of binary classification problems, one per emotion, similarly to mohammad et al. (2018) (this allows for the theoretical case where there are multiple emotions in one instance). we use logistic regression and multilingual bert as classifiers, to probe for various lexical and syntactic properties of the task. ## results we treat each emotion as a separate binary task, and compute a macro-average of the f1-scores for each of the six tasks. a random baseline model ( ## analysis of cross-lingual effects our results show a wide range of cross-lingual effects. in many cases, they are quite substantial, while in other cases we observe no effect. we believe these differences are due to the linguistic properties of the languages that the models pick up on. here we examine some of the factors involved in this: code switching, typological closeness, and linguistic diversity. the results can shed light on the similarities and differences in how emotions are expressed in different languages. ## conclusion we introduce a new data set of over 530,000 anonymized facebook posts from 18 languages, labeled with five basic emotions. we show that emotions can be reliably identified, both within and across languages, including zero-shot learning. this suggests substantial opportunities for transferring knowledge from high-resource to low-resource languages. in a detailed investigation of the factors supporting cross-lingual learning, we find evidence for the importance of linguistic diversity of training data as well as syntactic and typological similarities between languages. these results provide intriguing evidence of deep commonalities in the linguistic expression of emotion across the languages of the world.