Enhancing Cognitive Models of Emotions with Representation Learning (2021)

Abstract: We present a novel deep learning-based framework to generate embedding representations of fine-grained emotions that can be used to computationally describe psychological models of emotions. Our framework integrates a contextualized embedding encoder with a multi-head probing model that enables to interpret dynamically learned representations optimized for an emotion classification task. Our model is evaluated on the Empathetic Dialogue dataset and shows the state-of-the-art result for classifying 32 emotions. Our layer analysis can derive an emotion graph to depict hierarchical relations among the emotions. Our emotion representations can be used to generate an emotion wheel directly comparable to the one from Plutchik’s model, and also augment the values of missing emotions in the PAD emotional state model.

## introduction emotion classification has been extensively studied by many disciplines for decades @xcite @xcite . two main streams have been developed for this research: one is the discrete theory that tries to explain emotions with basic and complex categories @xcite @xcite , and the other is the dimensional theory that aims to conceptualize emotions into a continuous vector space @xcite @xcite . illustration of human emotion however is often subjective and obscure in nature, leading to a long debate among researchers about the "correct" way of representing emotions @xcite . representation learning has made remarkable progress recently by building neural language models on large corpora, which have substantially improved the performance on many downstream tasks @xcite @xcite @xcite . encouraged by this rapid progress along with an increasing interest of interpretability in deep learning models, several studies have attempted to capture various knowledge encoded in language @xcite @xcite , and shown that it is possible to learn computational representations through distributional semantics for abstract concepts. inspired by these prior studies, we build a deep learning-based framework to generate emotion embeddings from text and assess its ability of enhancing cognitive models of emotions. our contributions are summarized as follows: foot_0 ## related work probing models are designed to construct a probe to detect knowledge in embedding representations. @xcite used linear probes to examine phrasal information in representations learned by deep neural models on multiple nlp tasks. @xcite proposed an edge probing model using a span pooling to analyze syntactic and semantic relations among words through word embeddings. @xcite constructed a structural probe to detect the correlations among word pairs to predict their latent distances in dependency trees. as far as we can tell, our work is the first to generate embeddings of fine-grained emotions from text and apply them to well-established emotion theories. nlp researchers have produced several corpora for emotion detection including friendsed @xcite , emoint @xcite , emobank @xcite , and daily-dialogs @xcite , all of which are based on coarse-grained emotions with at most 7 categories. for a more comprehensive analysis, we adapt the empathetic dialogue dataset based on fine-grained emotions with 32 categories @xcite . ## multi-head probing model we present a multi-head probing model allowing us to interpret how emotion embeddings are learned in deep learning models. figure 1 shows an overview of our probing model. let @xmath0, . . . , w n } be an input document where w i is the i'th token in the document. w is first fed into a contextualized embedding encoder that generates the embedding e 0 ∈ r d 0 representing the entire document. the document embedding e 0 is then fed into multiple probing heads, ph 11 , . . . , ph 1k , that generate the vectors e 1j ∈ r d 1 comprising features useful for emotion classification (@xmath1, k]). the probing heads in this layer are expected to capture abstract concepts (e.g., positive/negative, intense/mild). each vector e 1j is fed into a sequence of probing heads where the probing head ph ij is defined the feature vectors e ℓ * from the final probing layer are expected to learn more fine-grained concepts (e.g., ashamed/embarrassed, hopeful/anticipating). e ℓ * are concatenated and normalized to g ℓ ∈ r d ℓ •k and fed into a linear layer that generates the output vector @xmath2. it is worth mentioning that every probing sequence finds its own feature combinations. thus, each of e ℓ * potentially represents different concepts in emotions, which allow us to analyze concept compositions of these emotions empirically derived by this model. ## conclusion this paper presents a multi-head probing model to derive emotion embeddings from neural model interpretation. our model is applied to an emotion detection task and shows a state-of-the-art result. these emotion embeddings can derive an emotion graph, depicting how abstract concepts are learned in neural models, and an emotion wheel and pad values, verifying their potential of augmenting cognitive models for more diverse groups of emotions that have not been explored by cognitive theories. by our regressions models, which are plotted in figure 4. finally, figure 7 plots those predicted pad values in the 3d space to depict the dominance values with respect to the other two pa dimensions. by comparing the pad values of 22 emotions in