Towards Label-Agnostic Emotion Embeddings (2021)

Abstract: Research in emotion analysis is scattered across different label formats (e.g., polarity types, basic emotion categories, and affective dimensions), linguistic levels (word vs. sentence vs. discourse), and, of course, (few well-resourced but much more under-resourced) natural languages and text genres (e.g., product reviews, tweets, news). The resulting heterogeneity makes data and software developed under these conflicting constraints hard to compare and challenging to integrate. To resolve this unsatisfactory state of affairs we here propose a training scheme that learns a shared latent representation of emotion independent from different label formats, natural languages, and even disparate model architectures. Experiments on a wide range of datasets indicate that this approach yields the desired interoperability without penalizing prediction quality. Code and data are archived under DOI 10.5281/zenodo.5466068.

## introduction emotion analysis in the field of nlp foot_0 has experienced a remarkable evolution of representation schemes. starting from the early focus on polarity, i.e., the main distinction between positive and negative feelings emerging from natural language utterances @xcite , the number and variety of label formats, i.e., groups of emotional target variables and their associated value ranges, has been growing rapidly @xcite . this development is a double-edged sword though. on the one hand, the wide variety of available label formats allows nlp models to become more informative and richer in expressive power. this gain is because many of the newer representation schemes follow well-researched branches of psychological theory, such as basic emotion categories or affective dimensions @xcite , which offer information complementary to each other @xcite . others argue that different emotional nuances turn out to be particularly useful for specific targeted downstream applications @xcite . on the other hand, this proliferation of label formats has led to a severe loss in cross-data comparability. as tab. 1 illustrates, the total volume of available gold data is spread not only over distinct languages but also a huge number of emotion annotation schemes. consequently, comparing or even merging data from different rating studies is often impossible. this, in turn, contributes to the development of an unnecessarily large number of prediction models, each with limited coverage of the full range of human emotion. to escape from these dilemmata, we propose a method that mediates between such different representation schemes. in contrast to previous work which unified some sources of heterogeneity (see §2), to the best of our knowledge, our approach is the first to learn a representation space for emotions that generalizes over individual languages, emotion label formats, and distinct model architectures for emotion analysis. technically speaking, our approach consists of a set of pre-trained prediction heads that can be easily attached to existing state-of-the-art neural models. doing so, a model learns to embed language items of a particular domain in a shared representation space that resembles an "interlingua for emotion". these "emotion embeddings" capture a rich array of affective nuances and allow for a direct comparison of emotional load between heterogeneous samples (see fig. 1 ). they may thus form a solid basis for a broad range of linguistic, psychological, and cultural follow-up studies. ## methods let (x, y ) be a dataset with samples x:={x 1 , . . . x n } and labels y :={y 1 , . . . , y n }. the aim of emotion analysis is to find a model f that best predicts y given x. let us assume that the samples x are drawn from one of m domains d 1 , . . . , d m and the labels are drawn from one of n label formats l 1 , . . . , l n . a domain refers to the vocabulary or a particular register of a given language (word-and text-level prediction). a label format is a set of valid labels with reference to particular emotion constructs. for instance, the vad format consists of vectors (v, a, d) where the components v, a, d refer to valence, arousal, and dominance, respectively, and are bound within a specified interval, e.g., @xcite . ## experimental setup the main idea behind our experimental setup is to compare a base model trained with the standard procedure against the same model with portable prediction heads (pph) attached (cf. fig. 2 (a) vs. (c)). our goal is to show that we obtain the same, if not better, results using pph compared with the naïve approach. this study design reflects two purposes. first, comparing the base model with the pph architecture yields experimental data that allow to indirectly assess the quality of the learned emotion representations. second, such a comparison may help find evidence that the performance of the pph approach scales with the employed base model-this would suggest that our method is likely to remain valuable even when today's state-of-the-art models are replaced by their successors. importantly, we train only a single set of prediction heads. thus, all experimental results of the pph condition are based on the same underlying emotion space. we distinguish two evaluation settings. in the first ("supervised") setting, train and test data come from (different parts of) the same dataset. without pph, we train one base model per dataset. yet, with pph, base models are shared across datasets of the same domain, whether or not their label formats agree. consequently, the emotion space needs to store heterogeneous affective information in an easy-to-retrieve way (recall the "lightweight" head design; §3.2). thus, positive evaluation results would indicate that our method learns a particularly rich representation of emotion. a practical advantage of pph lies in the reduction of total disk space utilized by the resulting model checkpoints. the second ("zero-shot") setting assumes that only one dataset per language is available, with one particular label format, but one would like to predict ratings in another format as well (e.g., imagine having a va dataset for mandarin but you are actually more interested in basic emotions for that language). doing so with pph is very simple-one only has to choose the desired head at inference time. yet, doing so with the base model per se is simply impossible. to still be able to offer a quantitative comparison, we resort to an external label mapping component that translates the base model's output into the desired format. we emphasize that this is a very strong baseline due to the high accuracy of the label mapping approach, in general @xcite . in this case, the practical advantage of the pph approach lies in its independence of (possibly unavailable) external post-processors. we conducted experiments on different word and text datasets. for words, we collected ten datasets (cf. tab. 2) covering five languages. these data are structured as illustrated in the top half of tab. 1 . for text-level experiments we selected three corpora (cf. tab. 3): affective text (afft; @xcite , emobank (emob; @xcite , and the chinese valence arousal texts (cvat; @xcite . for an illustration of the type and format of text-level data, see the bottom half in tab. 1. since these datasets comprise real-valued annotations, we will use pearson correlation r for measuring prediction quality. datasets were partitioned into fixed train-dev-test splits with ratios ranging between 8-1-1 and 3-1-1; smaller datasets received larger dev and test shares. the selected data govern how to train a given base model with pph ( §3.3). since, except for mandarin, there are always two datasets available per domain, we train the models in the supervised setting using the multi-task approach (but use emotion label augmentation for cvat). by contrast, in the zero-shot setting, we train a model on one, yet test on another dataset. thus, we rely on emotion label augmentation here (and have to exclude cvat for a lack of a second mandarin dataset). we emphasize that the zero-shot evaluation has very demanding data requirements: this setting not only requires two datasets of the same language domain with different label formats (which is already rare) but also additional data to fit mapping models for those particular label formats. to the best of our id vars size citation en1 vad 1,034 bradley and lang (1999) en2 be5 1,034 stevenson et al. (2007) es1 va 14,031 stadthagen-gonzález et al. (2017) es2 be5 10,491 stadthagen-gonzález et al. (2018) de1 va 2,902 võ et al. (2009) de2 be5 1,958 briesemeister et al. (2011) pl1 va 2,902 riegel et al. (2015) pl2 be5 2,902 wierzba et al. (2015) tr1 va 2,029 kapucu et al. (2018) tr2 be5 2,029 kapucu et al. (2018) importantly, only the data requirements for evaluating our approach in the zero-shot setting are hard to meet. yet, inference is much easier to provide. we would even argue that the reason why our method is so hard to evaluate is precisely what makes it so valuable. take the mandarin cvat dataset, for example. it is annotated with valence and arousal, but there is, to our knowledge, no compatible mandarin dataset with basic emotions (thus, cvat is not used in the zero-shot setting). our method allows to freely switch between output label formats at inference time without language constraints. that is, we can predict be5 ratings in chinese even though there is no such training data. in terms of base models, we used the feed-forward network developed by @xcite for the word datasets. this model predicts emotion ratings based on pre-trained embedding vectors (taken from @xcite . for text datasets, we chose the bert base transformer model by @xcite using the implementation and pre-trained weights by @xcite . both (word and text) base models use identical hyperparameter settings with or without pph extension. for the word model, we copied the settings of the authors, whereas text model hyperparameters were tuned manually for the base model without pph. we derived training data for the prediction heads (label mapping datasets) by combining the ratings of the word datasets en1 and en2. we used the label mapping model from @xcite as auxiliary label encoders. the dimensionality of the emotion space was set to 100. the label mapping models used as external post-processors in the zero-shot setting were also based on @xcite and were trained on the same data as the label encoders. further details beneficial for reproducibility are given in appendix d. ## results our main experimental results are summarized in tables 4 to 7 . for conciseness, correlation values are averaged over all target variables per dataset. per-variable results are given in appendix b. looking at the word datasets in the supervised setup (tab. 4), we find that attaching portable prediction heads (pph) not only retains, but often enough slightly increases the performance of the ffn base model (p=.008; two-sided wilcoxon signed-rank test based on per-dataset results). since we trained only one base model with pph per language (but two without pph), our data suggest that the emotion representations learned with pph can easily hold affective information from different label formats at the same time. moreover, pph here offers the practical benefit of reducing the total disk space used by the resulting model checkpoints due to the smaller number of trained base models. experiments on the text datasets using bert as base model show results in line with these findings (see tab. 5 ). in the zero-shot setup, models are tested on datasets with label formats different from the training phase (e.g., en1 and en2). on the word datasets, using pph shows small improvements in comparison with the base model as is (p=.003; tab. 6), again suggesting that the learned emotion representations generalize robustly across label formats. importantly, the base model is only capable of producing this label format at all because we equip it with a label mapping post-processor. while this procedure is very accurate (indeed, it constitutes a very strong baseline), it depends on an external component that may or may not be available for base model (ffn) base model + pph test data train data r train data r en1(vad) en1(vad) .818 en1+en2 .824 en2(be5) en2(be5) .898 en1+en2 .898 es1(va) es1(va) .820 es1+es2 .833 es2(be5) es2(be5) .789 es1+es2 .820 de1(va) de1(va) .822 de1+de2 .836 de2(be5) de2(be5) .754 de1+de2 .748 pl1(va) pl1(va) .794 pl1+pl2 .835 pl2(be5) pl2(be5) .814 pl1+pl2 .845 tr1(va) tr1(va) .567 tr1+tr2 .575 tr2(be5) tr2(be5) .607 tr1+tr2 .614 mean .768 .783 disk use 4.33 mb 2.52 mb the desired mapping direction (the source and the target label format). in contrast, the zero-shot capability is innate to ("built-in") the pph approach. while we need only one prediction head per label format, the number of required mapping components for the base model grows on a quadratic scale with the number of considered formats. again, text-level experiments show consistent results with word-level ones (tab. 7). one may object that the reduction of memory footprint shown in tables 4 and 5 can also be achieved by traditional multi-task learning (i.e., attaching multiple heads to the base model, training it on two datasets, at once). likewise, as tables 6 and 7 indicate, the zero-shot capabilities offered by pph can, in principle, be provided by additional label mapping components. however, pph offers a much more elegant solution to combine the advantages of multi-task learning and label mapping without calling for additional (language) resources. most importantly though, pph is unique in its ability to embed samples from such heterogeneous datasets in a common representation space-a trait that may offer a general solution to studying emotion across languages, cultures, and individually preferred psychological theory. ## visualization of the emotion space to gain first insights into the structure of our learned emotion space, we submitted the weight vectors of the emotion variables to principal com-ponent analysis (pca; recall from §3.2 that each row in a head's weights matrix w corresponds to exactly one variable). further, we derived emotion embeddings for the samples in tab. 1 using the pph-extended models evaluated in the last section. applying the same pca transformation to the embedding vectors, we co-locate the samples next to the emotion variables. the results (for the first three pcs) are displayed in fig. 1 . as can be seen, the relative positioning of the samples and variables shows high face validity-samples associated with similar feelings appear close to each other as well as to their akin variable. appendix c provides additional analyses of the learned embedding space (focusing more deeply on the emotional interpretation of the pc axes and the distribution of emotion embeddings across languages) that further support this positive impression. ## conclusions & future work we presented a method for learning a common representation space for the emotional loading of heterogeneous language items. while previous work successfully unified some sources' heterogeneity, our emotion embeddings are the first to comprehensively generalize over arbitrarily disparate language domains, label formats, and distinct neural network architectures. our technique is based on a collection of portable prediction heads that can be attached to existing state-of-the-art models. consequently, a model learns to embed language items in the common learned emotion space and thus to predict a wider range of emotional meaning facets, yet without sacrificing any predictive power as our experiments on 13 datasets (6 languages) indicate. since the resulting emotion representations both generalize across various use cases and evidently capture a rich set of affective nuances, we consider this work particularly useful for downstream applications. thus, future work may build on a concept of emotion similarity to, e.g., cluster diverse language items by their associated feeling, retrieve words that evoke emotions similar to a query, or compare the affective meaning of phrases and concepts across cultures.