Topic and Style-aware Transformer for Multimodal Emotion Recognition (2023)

Abstract: Understanding emotion expressions in multimodal signals is key for machines to have a better understanding of human communication. While language, visual and acoustic modalities can provide clues from different perspectives, the visual modality is shown to make minimal contribution to the performance in the emotion recognition field due to its high dimensionality. Therefore, we first leverage the strong multimodality backbone VATT to project the visual signal to the common space with language and acoustic signals. Also, we propose content-oriented features Topic and Speaking style on top of it to approach the subjectivity issues. Experiments conducted on the benchmark dataset MOSEI show our model can outperform SOTA results and effectively incorporate visual signals and handle subjectivity issues by serving as content “normalization”.

## introduction emotion recognition is intrinsic for social robots to interact with people naturally. the ability to tell emotional change and propose timely intervention solutions can help maintain people's mental health and social relations. though the traditional task of sentiment analysis is purely based on text @xcite @xcite , humans express emotions not only with spoken words but also through non-verbal signals such as facial expressions and the change of tones. therefore, following the current trend of multimodal emotion recognition @xcite @xcite , we focus on addressing problems of understanding the expressed emotions in videos along with their audio and transcripts. in this work, we tackle the problem of the multimodal emotion recognition task from two major issues: minimal contribution of visual modality, and emotional subjectivity. previous works which have used multimodal approaches @xcite figure 1 : left table: "happy" under different topics. right table: speaking styles can affect how emotion is displayed on the face 2020; @xcite have shown that text+audio outperforms the results of combining all three modalities. while facial and gesture signals contain abundant information, they tend to introduce more noise to the data due to its high dimensionality. in order to increase the contribution from visual modality , we propose to take advantage of the strong multimodal backbone vatt @xcite that can project features of different granularity levels into a common space. on the other hand, the expression of emotion is subjective. people's emotion judgment can be influenced by enclosed scenarios. as shown in the left two columns in figure 1 , though the two examples are all labeled as "happy", the signals we use to detect "happy" may not be the same. in a public speech, showing gratitude may mean a positive sentiment while in movie reviews, we may focus more on sentiment words like good or bad. also, subjectivity may come from individual differences in their own emotional intensity. as the examples shown in the right three columns in figure 1 , the sadness and happiness of the person in the excited style are more distinguishable through his face while the person in the calm style always adopts a calm face that makes sad and happy less recognizable. therefore, we introduce content-oriented features: topic and speaking style serving as a content "normalization" for each person. our work makes the following contribution: ## related work emotion recognition using a fusion of input modalities such as text, speech, image, etc is the key research direction of human-computer interaction. specific to the area of sentiment analysis, multimodal transformer applies pairwise crossattention to different modalities @xcite . the memory fusion network synchronizes multimodal sequences using a multi-view gated memory that stores intra-view and cross-view interactions through time @xcite . tfn performs the outer product of the modalities to learn both the intra-modality and inter-modality dynamics @xcite . @xcite begins the endeavor to take bert @xcite as a strong backbone pretrained on large scale corpus. @xcite follows the direction and combines roberta with a light-weighed audio encoder to fuse the text and audio features. a recent work @xcite presents a self-supervised framework to pretrain features within a single modality and across different modalities. other frameworks include context and speaker-aware rnn @xcite , graph neural networks modeling knowledge graphs and inter/intra relations between videos @xcite @xcite , while @xcite has used topic information to improve emotion detection 3 method ## limitations for modeling simplicity, we adopt the classic lda methods to get the topic id for each video segment. we plan to investigate more advanced topic clustering methods and check how it can be applied to multilingual cases. also, we propose a twostage framework that first extract topic and style features, based on which the emotion classifier will be trained. in the future, we hope to extend this work to learn features in an end-to-end manner.