How Language-Dependent is Emotion Detection? Evidence from Multilingual BERT (2022)

Abstract: As emotion analysis in text has gained a lot of attention in the field of natural language processing, differences in emotion expression across languages could have consequences for how emotion detection models work. We evaluate the language-dependence of an mBERT-based emotion detection model by comparing language identification performance before and after fine-tuning on emotion detection, and performing (adjusted) zero-shot experiments to assess whether emotion detection models rely on language-specific information. When dealing with typologically dissimilar languages, we found evidence for the language-dependence of emotion detection.

## introduction as language finds itself at the crossroads of cognition and culture, it has been a thoroughly investigated subject in the context of emotion research and has given rise to questions as how emotion expression varies across languages and whether language has an impact on emotion conceptualisation and perception. indeed, many studies have reported on the cultural relativity of emotion, often underscoring the diversity in emotion lexicons across languages: not only is there a big variability in which emotional states are included in the lexicon of a language with a designated emotion term (e.g., a word for sadness seems to be missing in tahiti @xcite ), but there are also many differences in the connotations and meanings of emotion terms across languages @xcite . instead of focusing on emotion conceptualisation and experience, one could also ask whether emotions are expressed differently across languages. again, this can be reflected in differences in emotion vocabulary, but also in languagespecific phraseology. in russian, for example, the * these authors contributed equally to this work. verbalisation of emotion is very much focused on the human body, and the numerous diminutive suffixes exhibit different emotional nuances @xcite . noteworthy is also the distinction between individualistic and collectivist cultures, where the latter are associated with more reticence to express emotions, while the former exhibit more open emotion expression @xcite . as emotion analysis in text has gained a lot of attention in artificial intelligence and the field of natural language processing (nlp) as well @xcite , languagedependent conceptualisation and expression could have consequences for how emotion detection models work. analogously to humans who might need knowledge about the linguistic code (e.g., to know whether irony is often used in a specific language or to understand language-specific phraseology) to correctly judge the emotional value of someone's utterance, machine learning models might need this knowledge as well in order to accurately predict emotions from text. therefore, we investigate the language-dependence of the task of emotion detection. in other words, we want to know whether knowledge about the language identity is needed to make accurate emotion predictions. for this analysis, we will look at languages from different language families and branches (e.g., germanic, italic and indo-iranian in the indo-european language family or chinese from the sino-tibetan language family) in order to include languages with different structural features. although language families are not the same as the classes defined in the field of linguistic typology (i.e., the analysis, comparison, and classification of languages according to their common structural features and forms), languages within one language family are generally more typologically similar than languages from different families. as transformer models are currently state of the art in many nlp tasks, we investigate the languagedependence of multilingual bert (mbert), the transformer model introduced by @xcite which was trained on 104 languages. we foresee two kinds of experiments. first, we investigate how much language-specific information is preserved in the bert representations by comparing performance on the task of language identification both before and after fine-tuning on emotion detection. second, zero-shot transfer learning (training on a source language and testing directly on the target language english) is compared with training on machine-translated data, i.e., data that was originally in english but automatically translated to the source language ('semi-zero-shot transfer learning'). these models thus learn from the same source language, but in the semi-zero-shot set-up language-specific information from the target language (like idioms, phraseology or cultural codes) might still be preserved, thus aiding performance during test time on the target language. in section 2, we describe the literature on crosslingual emotion research (section 2.1) and discuss related work dealing with language dependency in nlp (section 2.2). in section 3 we explain our method by describing the data and resources (section 3.1) and by zooming in on the experimental set-up (section 3.2). the results are reported in section 4 and further discussed in section 5, followed by a conclusion in section 6. ## discussion although we found some potential evidence for the language-dependence of emotion detection, several points need to be taken into account. first of all, the datasets used in this study are small (especially for the category fear), and the overall quality of the data is low. it seems that some messages are incomplete and that some (parts of) instances appear multiple times in the dataset. 3 furthermore, some instances contain code-switching between different languages. another drawback is that we only tested on english. we made this choice because we could not obtain test sets for all languages (for hindi and dutch, all data was already used for training). we claim that we found evidence for the language-dependence of emotion detection, where typologically dissimilar languages suffer more from cross-lingual zero-shot learning. this evidence is partly based on the observation that semizero-shot experiments (in which language-specific information is assumed to be preserved to a certain extent) outperforms zero-shot learning for hindi and chinese, while it does not for the european languages (as language-specific information might be similar for these languages and english, and there therefore is no benefit in using a model that encodes this information). however, it could be that this language-specific information is not related to phraseology or differences in emotion-topic relations (see section 1 and 2.1), but to differences in topic distribution in general. it might be the case that the topics in chinese and hindi are very different from the topics in the english dataset, while the european languages contain similar topic distributions as english. the semi-zero-shot experiments are based on the idea that some language-specific information is preserved after machine translation. although we cannot be absolutely certain of this, the fact that the semi-zero-shot experiments outperformed normal zero-shot for some languages, suggests that there is some helpful information in these translations. one could argue that the performance of the semi-zeroshot models correlates negatively with the quality of the translations: machine translation might be bad for less similar languages, resulting in a better emotion classification performance in the semizero-shot case, because some words have not been translated. however, we could not find evidence for this. when applying a token-level language identifier on the translated texts 4 , we found that the percentage of tokens that was classified as english instead of chinese, tagalog and hindi is respectively 6%, 3% and 0.3%. that there are almost no untranslated words in the hindi set while the semizero-shot does perform better, thus contradicts that the semi-zero-shot performance is explained by the number of untranslated words. in future work, we envisage to use a different approach for investigating the language-dependence of emotion detection instead of relying on semizero-shot experiments. as both this study and previous research has shown that mbert partitions its representations per language @xcite , it would be compelling to see whether we can achieve language-neutral representations and which effect that has on the emotion detection performance. we hypothesise that when the representations no longer exhibit language-specific information, it would hamper emotion detection. ## conclusion in this paper, we assessed the language-dependence of an mbert-based emotion detection model. we first investigated the effect of fine-tuning on emotion on the preservation of language-specific information in mbert, by comparing language identification performance of the languages chinese, dutch, english, hindi, portuguese, spanish and tagalog before and after fine-tuning on emotion detection and visualising the model's hidden states in t-sne plots. as expected, language-specific information is lost after fine-tuning, but only to a small extent. especially the representations of typologically dissimilar languages remain more or less isolated, while similar languages get clustered together. in a next set of experiments, we compared zeroshot learning with what we called 'semi-zero-shot learning'. in the zero-shot experiments, we trained a model on either chinese, dutch, hindi, portuguese, spanish or tagalog and tested it on english data. in semi-zero-shot, originally english data was translated to those languages, assuming that some language-specific information is preserved in these translations. we found that for the european languages, normal zero-shot is better than semi-zero shot. however, for less similar languages, semi-zero-shot was better, suggesting that there is some language-specific information aiding the performance. this could be evidence for the language-dependence of emotion detection. future research, dealing with better datasets and approaches to make the bert representations language-neutral, should be carried out to corroborate these findings.