Learning and Evaluating Emotion Lexicons for 91 Languages (2020)

Abstract: Emotion lexicons describe the affective meaning of words and thus constitute a centerpiece for advanced sentiment and emotion analysis. Yet, manually curated lexicons are only available for a handful of languages, leaving most languages of the world without such a precious resource for downstream applications. Even worse, their coverage is often limited both in terms of the lexical units they contain and the emotional variables they feature. In order to break this bottleneck, we here introduce a methodology for creating almost arbitrarily large emotion lexicons for any target language. Our approach requires nothing but a source language emotion lexicon, a bilingual word translation model, and a target language embedding model. Fulfilling these requirements for 91 languages, we are able to generate representationally rich high-coverage lexicons comprising eight emotional variables with more than 100k lexical entries each. We evaluated the automatically generated lexicons against human judgment from 26 datasets, spanning 12 typologically diverse languages, and found that our approach produces results in line with state-of-the-art monolingual approaches to lexicon creation and even surpasses human reliability for some languages and variables. Code and data are available at https://github.com/JULIELab/MEmoLon archived under DOI 10.5281/zenodo.3779901.

## introduction an emotion lexicon is a lexical repository which encodes the affective meaning of individual words (lexical entries). most simply, affective meaning can be encoded in terms of polarity, i.e., the distinction whether an item is considered as positive, negative, or neutral. this is the case for many well-known resources such as wordnet-affect @xcite , sentiword-net @xcite , or vader @xcite ). yet, an increasing number of researchers focus on more expressive encodings for affective states inspired by distinct lines of work in psychology @xcite @xcite @xcite @xcite . psychologists, on the one hand, value such lexicons as a controlled set of stimuli for designing experiments, e.g., to investigate patterns of lexical access or the structure of memory @xcite . nlp researchers, on the other hand, use them to augment the emotional loading of word embeddings @xcite , as additional input to sentence-level emotion models so that the performance of even the most sophisticated neural network gets boosted @xcite @xcite , or rely on them in a keyword-spotting approach when no training data is available, e.g., for studies dealing with historical language stages @xcite . as with any kind of manually curated resource, the availability of emotion lexicons is heavily restricted to only a few languages whose exact number varies depending on the variables under scrutiny. for example, we are aware of lexicons for 15 languages that encode the emotional variables of valence, arousal, and dominance (see section 2). this number leaves the majority of the world's (less-resourced) languages without such a dataset. in case such a lexicon exists for a particular language, it is often severely limited in size, sometimes only comprising some hundreds of entries @xcite ). yet, even the largest lexicons typically cover only some ten thousands of words, still leaving out major portions of the emotion-carrying vocabulary. this is especially true for languages with complex morphology or productive compounding, such as finnish, turkish, czech, or german. finally, the diversity of emotion representation schemes adds another layer of complexity. while psychologists and nlp researchers alike find that different sets of emotional variables are complementary to each other @xcite @xcite , manually creating emotion lexicons for every language and every emotion representation scheme is virtually impossible. we here propose an approach based on crosslingual distant supervision to generate almost arbitrarily large emotion lexicons for any target language and emotional variable, provided the following requirements are met: a source language emotion lexicon covering the desired variables, a bilingual word translation model, and a target language embedding model. by fulfilling these preconditions, we can automatically generate emotion lexicons for 91 languages covering ratings for eight emotional variables and hundreds of thousands of lexical entries each. our experiments reveal that our method is on a par with state-of-the-art monolingual approaches and compares favorably with (sometimes even outperforms) human reliability. another dividing line between representational approaches is whether target variables are encoded in terms of (strict) class-membership or scores for numerical strength. in the first case, emotion analysis translates into a (multi-class) classification problem, whereas the latter turns it into a regression problem @xcite . while our proposed methodology is agnostic towards the chosen emotion format, we will focus on the vad and be5 formats here, using numerical ratings (see the examples in ## a novel approach to lexicon creation our methodology integrates (1) cross-lingual generation and expansion of emotion lexicons and (2) their evaluation against gold and silver standard data. consequently, a key aspect of our workflow design is how data is split into train, dev, and test sets at different points of the generation process. figure 1 gives an overview of our framework including a toy example for illustration. lexicon generation. we start with a lexicon (source) of arbitrary size, emotion format foot_0 and source language which is partitioned into train, dev, and test splits denoted by source-train, source-dev, and source-test, respectively. next, we leverage a bilingual word translation model between source and desired target language to build the first target-side emotion lexicon denoted as targetmt. source words are translated according to the model, whereas target-side emotion labels are simply copied from the source to the target (see section 2). entries are assigned to train, dev, or test set according to their source-side assignment (cf. figure 1 ). the choice of our translation service (see below) ensures that each source word receives exactly one translation. targetmt is then used as the distant supervisor to train a model that predicts word emotions based on target-side word embeddings. targetmt-train and targetmt-dev are used to fit model parameters and optimize hyperparameters, respectively, whereas targetmt-test is held out for later evaluation. once finalized, the model is used to predict new labels for the words in targetmt, resulting in a second target-side emotion lexicon denoted targetpred. our rationale for doing so is that a reasonably trained model should generalize well figure 1: schematic view on the methodology for generating and evaluating an emotion lexicon for a given target language based on source language supervision. included is a toy example starting with an english va lexicon (sunshine, nuclear, terrorism and the associated numerical scores for valence and arousal) and resulting in an extended german lexicon which incorporates translated entries with altered va scores and additional entries originating from the embedding model with newly learned scores. the splits are defined as follows: let m t train , m t dev , and m t test denote the set of words in train, dev, and test split of targetmt, respectively. likewise, let p train , p dev , and p test denote the splits of targetpred and let e denote the set of words in the embedding model. then the above definitions help clarify the way we address polysemy. 2 ambiguity on the target-side 2 in short, our work evades this problem by dealing with lexical entries exclusively on the type-rather than the senselevel. from a lexicological perspective, this may seem like a strong assumption. from a modeling perspective, however, it appears almost obvious as it aligns well with the major components of our methodology, i.e., lexicons, embeddings, and translation. the lexicons we work with follow the design of behavioral experiments: a stimulus (word type) is given to may result in multiple source entries translating to the same target-side word. 3 this circumstance leads to "partial duplicates" in targetmt, i.e., groups of entries with the same word type but different emotion values (because they were derived from distinct source entries). such overlap could do harm to the integrity of our evaluation since knowledge may "leak" from training to validation phase, i.e., by testing the model on words it has already seen during training, although with distinct emotion labels. the proposed data partitioning eliminates such distortion effects. since partial duplicates receive the same embedding vector, the prediction model assigns the same emotion value to both, thus merging them in targetpred. evaluation methodology. the main advantage of the above generation method is that it allows us to create large-scale emotion lexicons for languages a subject and the response (rating) is recorded. the absence of sense-level annotation simplifies the mapping between lexicon and embedding entries. while sense embeddings form an active area of research @xcite , to the best of our knowledge, type-level embeddings yield state-of-the-art performance in downstream applications. for which gold data is lacking. but if that is the case, how can we assess the quality of the generated lexicons? our solution is to propose two different evaluation scenarios-a gold evaluation which is a strict comparison against human judgment, meaning that it is limited to languages where such data (denoted targetgold) is available, and a silver evaluation which substitutes human judgments by automatically derived ones (silver standard) which is feasible for any language in our study. the rationale is that if both, gold and silver evaluation, strongly agree with each other, we can use one as proxy for the other when no target-side gold data exists (examined in section 6). note that our lexicon generation approach consists of two major steps, translation and prediction. however, these two steps are not equally important for each generated entry in targetpred. words, such as german sonnenschein for which a translational equivalent already exists in the source ("sunshine"; see figure 1 ), mainly rely on translation, while the prediction step acts as an optional refinement procedure. in contrast, the prediction step is crucial for words, such as erdbeben, whose translational equivalents ("earthquake") are missing in the source. yet, these words also depend on the translation step for producing training data. these considerations are important for deciding which words to evaluate on. we may choose to base our evaluation on the full targetpred lexicon, including words from the training set-after all, the word emotion model does not have access to any target-side gold data. the problem with this approach is that it merges words that mainly rely on translation, because their equivalents are in the source, and those which largely depend on prediction, because they are taken from the embedding model. in this case, generalizability of evaluation results becomes questionable. thus, our evaluation methodology needs to fulfill the following two requirements: (1) evaluation must not be performed on translational equivalents of the source entries to which the model already had access during training (e.g., sonnenschein and nuklear in our example from figure 1 ); but, on the other hand, (2) a reasonable number of instances must be available for evaluation (ideally, as many as possible to increase reliability). the intricate cross-lingual train-dev-test set assignment of our generation methodology is in place so that we meet these two requirements. in particular, for our silver evaluation, we intersect targetmt-test with targetpred-test and compute the correlation of these two sets individually for each emotion variable. pearson's r will be used as correlation measure throughout this paper. establishing a test set at the very start of our workflow, source-test, assures that there is a relatively large overlap between the two sets and, by extension, that our requirements for the evaluation are met. the gold evaluation is a somewhat more challenging case, because we can, in general, not guarantee that the overlap of a targetgold lexicon with targetpred-test will be of any particular size. for this reason, the words of the embedding model are added to targetpred-test (see above), maximizing the expected overlap with targetgold. in practical terms, we intersect targetgold with targetpred-test and compute the variable-wise correlation between these sets, in parallel to the silver evaluation. a complementary strategy for maximizing overlap, by exploiting dependencies between published lexicons, is described below. ## experimental setup gold lexicons and data splits. we use the english emotion lexicon from @xcite as first part of our source dataset. this popular resource comprises about 14k entries in vad format collected via crowdsourcing. since manually gathered be5 ratings are available only for a subset of this lexicon @xcite , we add be5 ratings from @xcite who used emotion representation mapping (see section 2) to convert the existing vad ratings, showing that this is about as reliable as human annotation. as apparent from the previous section, a crucial aspect for applying our methodology is the design of the train-dev-test split of the source because it directly impacts the amount of words we can test our lexicons on during gold evaluation. in line with these considerations, we choose the lexical items which are already present in anew @xcite as source-test set. anew is the precursor to the version later distributed by @xcite ; it is widely used and has been adapted to a wide range of languages. with this choice, it is likely that a resulting targetpred-test set has a large overlap with the respective targetgold lexicon. as for the targetgold lexicons, we included every va(d) and be5 lexicon we could get hold of with more than 500 entries. this resulted in 26 datasets covering 12 quite diverse languages (see translation. we used the google cloud translation api foot_1 to produce word-to-word translation tables. this is a commercial service, total translation costs amount to 160 eur. api calls were performed in november 2019. embeddings. we use the fasttext embedding models from @xcite trained for 157 languages on the respective wikipedia and the respective part of commoncrawl. these resources not only greatly facilitate our work but also increase comparability across languages. the restriction to "only" 91 languages comes from intersecting the ones covered by the vectors with the languages covered by the translation service. models. since our proposed methodology is agnostic towards the chosen word emotion model, we will re-use models from the literature. in particular, we will rely on the multi-task learning feed-forward network (mtlffn) worked out by @xcite . this network constitutes the current state of the art for monolingual emotion lexicon creation (expanding an existing lexicon for a given language) for many of the datasets in the mtlffn has two hidden layers of 256 and 128 units, respectively, and takes pre-trained embedding vectors as input. its distinguishing feature is that hidden layer parameters are shared between the different emotion target variables, thus constituting a mild form of multi-task learning (mtl). we apply mtl to vad and be5 variables individually (but not between both groups), thus training two distinct emotion models per language, following the outcome of a development experiment. details are given in appendix a.2 together with the remainder of the model specifications. being aware of the infamous instability of neural approaches @xcite , we also employ a ridge regression model, an l 2 regularized version of linear regression, as a more robust, yet also powerful baseline @xcite . ## results the size of the resulting lexicons (a complete list is provided in ## methodological assumptions revisited this section investigates patterns in prediction quality across languages, validating design decisions of our methodology. translation vs. prediction. is it beneficial to predict new ratings for the words in targetmt rather than using them as final lexicon entries straight away? for each targetgold lexicon (cf. v a l a r o d o m j o y a n g s a d f e a d i s pred .871 .652 .733 .767 .734 .692 .728 .650 mt .796 .515 .613 .699 .677 .636 .654 .579 diff .076 .137 .119 .068 .057 .056 .074 .071 ## conclusion emotion lexicons are at the core of sentiment analysis, a rapidly flourishing field of nlp. yet, despite large community efforts, the coverage of existing lexicons is still limited in terms of languages, size, and types of emotion variables. while there are techniques to tackle these three forms of sparsity in isolation, we introduced a methodology which allows us to cope with them simultaneously by jointly combining emotion representation mapping, machine translation, and embedding-based lexicon expansion. our study is "large-scale" in many respects. we created representationally complex lexiconscomprising 8 distinct emotion variables-for 91 languages with up to 2 million entries each. the evaluation of the generated lexicons featured 26 manually annotated datasets spanning 12 diverse languages. the predicted ratings showed consistently high correlation with human judgment, compared favorably with state-of-the-art monolingual approaches to lexicon expansion and even surpassed human inter-study reliability in some cases. the sheer number of test sets we used allowed us to validate fundamental methodological assumptions underlying our approach. firstly, the evaluation procedure, which is integrated into the generation methodology, allows us to reliably estimate the quality of resulting lexicons, even without target language gold standard. secondly, our data suggests that embedding-based word emotion models can be used as a repair mechanism, mitigating poor target-language emotion estimates acquired by simple word-to-word translation. future work will have to deepen the way we deal with word sense ambiguity by way of exchanging the simplifying type-level approach our current work is based on with a semantically more informed sense-level approach. a promising direction would be to combine a multilingual sense inventory such as babelnet @xcite with sense embeddings @xcite . with a chinese native speaker for both of these procedures regarding the zh1 and zh2 lexicons.