E mo C aps: Emotion Capsule based Model for Conversational Emotion Recognition (2022)

Abstract: Emotion recognition in conversation (ERC) aims to analyze the speaker’s state and identify their emotion in the conversation. Recent works in ERC focus on context modeling but ignore the representation of contextual emotional tendency. In order to extract multi-modal information and the emotional tendency of the utterance effectively, we propose a new structure named Emoformer to extract multi-modal emotion vectors from different modalities and fuse them with sentence vector to be an emotion capsule. Furthermore, we design an end-to-end ERC model called EmoCaps, which extracts emotion vectors through the Emoformer structure and obtain the emotion classification results from a context analysis model. Through the experiments with two benchmark datasets, our model shows better performance than the existing state-of-the-art models.

## introduction emotion recognition in conversation (erc) is a work that recognizes the speaker's emotion and its influencing factors in the process of conversation. nowadays, social media such as facebook and twitter generate a large amount of dialogue data with various modalities of textual, audio, and video all the time. the study of speaker emotional tendency has huge potential value in the fields of public opinion analysis, shopping, and consumption. therefore, conversation emotion recognition has attracted more and more attention from researchers and companies. in erc, existing research mainly focuses on the way of contextual information modeling @xcite . however, these models have some shortcomings due to their inability to better extract the grammatical and semantic information of the utterance. recent studies @xcite have introduced the transformer structure into the utterance feature extraction to solve the above problems. @xcite proposed a new expression vector, "emotion vector" for erc, which is obtained by mapping from sentence vector, but only for textual modality. meanwhile, existing studies @xcite @xcite have shown that only textual information is not enough for emotional presentation, the tone and intonation reflect the speaker's emotions to a certain extent, and the facial expressions also express the inner feelings of the speaker in most cases. as shown in figure 1 , different modalities contain different information, and all are slightly flawed, so multi-modal based information can better identify the speaker's emotion than a single modality in erc. in order to identify the speaker's emotion in conversation effectively, it is necessary to obtain good utterance features. also we can't ignore the role of the utterance's emotional tendency. as shown in figure 2 , the emotional tendency of the utterance itself is like an "offset vector", which makes the neutral utterance have an "emotional direction". for single-sentence emotion classification, emotional tendency is consistent with the results of emotion recognition, while in erc, the influence of the context may cause the emotional tendency to be inconsistent with the result of emotion recognition. however, emotional tendency can provide features for the model so that model can "understand" the reason for emotional reversal. so, we propose a new multi-modal emotional tendency extraction method called emoformer, which is a transformer-based model but doesn't include the decoder part. as shown in figure 3 , emoformer extracts the emotional tendency, i.e., emotion vector, from the modal features through the multi-head self-attention layer and feed-forward layer. more details we will analysis in section 3. based on the emoformer, we further propose an end-to-end erc model to classify the emotion based on multi-modal information, named emo-caps. specifically, we employ the emoformer structure to extract emotion vectors of textual, audio, and visual features. then, we merge the emotion vectors of the three modalities with the sentence vector to an emotion capsule. finally, we employ a context analysis model to get the final result of the emotion classification. in general, the contributions of this paper are as follows: the rest of the paper is organized as follows: section 2 discusses related works; section 3 introduces the proposed emocaps model in detail; section 4 and 5 present the experiment setups on two benchmark datasets and the analysis of experiment results; finally, section 6 concludes the paper. ## results and analysis our proposed model is compared with other stateof-the-art models on the iemocap dataset and meld dataset, which are under the same parameter conditions. the experimental results are shown in ## conclusion in this paper, we propose a new multi-modal feature extraction structure, emoformer, which is based on the transformer structure. further, we design a new erc model, namely emocaps. first, we use emoformer structures to extract the emotion vectors of textual, audio, and visual modalities, then fuse the three modalities emotion vectors and sentence vectors to be an emotion capsule; finally, we employ a context analysis model to get the emotion recognition result. we conduct comparative experiments on two benchmark datasets. the experimental results show that our model performs better than the existing state-of-the-art models. the experimental results also verified the rationality of our hypothesis.