A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations (2023)

Abstract: Multimodal Emotion Recognition in Multiparty Conversations (MERMC) has recently attracted considerable attention. Due to the complexity of visual scenes in multi-party conversations, most previous MERMC studies mainly focus on text and audio modalities while ignoring visual information. Recently, several works proposed to extract face sequences as visual features and have shown the importance of visual information in MERMC. However, given an utterance, the face sequence extracted by previous methods may contain multiple people’s faces, which will inevitably introduce noise to the emotion prediction of the real speaker. To tackle this issue, we propose a two-stage framework named Facial expressionaware Multimodal Multi-Task learning (FacialMMT). Specifically, a pipeline method is first designed to extract the face sequence of the real speaker of each utterance, which consists of multimodal face recognition, unsupervised face clustering, and face matching. With the extracted face sequences, we propose a multimodal facial expression-aware emotion recognition model, which leverages the frame-level facial emotion distributions to help improve utterance-level emotion recognition based on multi-task learning. Experiments demonstrate the effectiveness of the proposed FacialMMT framework on the benchmark MELD dataset. The source code is publicly released at https://github.com/NUSTM/FacialMMT .

## introduction multimodal emotion recognition in multi-party conversations (mermc) is a challenging task in the field of multimodal research. the complexity of the task arises from the dynamic and spontaneous nature of human communication in multi-party conversations, which often involves multiple people expressing a variety of emotions simultaneously. in this task, the use of multiple modalities (e.g., meld @xcite ✗ ✗ ✗ unimse @xcite ✗ ✗ ✗ mmgcn @xcite ✓ ✗ ✗ mesm @xcite ✓ ✗ ✗ m 3 ed @xcite ✓ ✗ ✗ in the literature, most previous mermc studies primarily focus on the text and audio modalities @xcite @xcite , because the visual context in mermc often involves many people and complex environmental scenes, which may bring much noise to emotion recognition of the real speaker. owing to the indispensable role of visual modalities, a number of studies explored the potential of visual information in mermc @xcite @xcite , which employ 3d-cnns @xcite to extract video features and model the interaction and dependency between consecutive video frames. however, the visual information extracted by these methods still contains much noise from environmental scenes. to alleviate the visual noise from environmental scenes, several recent studies @xcite @xcite , 2022b) propose to detect all the faces in an utterance based on face detection tools such as mtcnn @xcite , openface @xcite or pre-trained active speaker detection models @xcite . however, given an utterance, the face sequence extracted by these methods may still contain multiple people, which may mislead the emotion prediction of the real speaker. for example, in figure 1 , there are two persons, joey and chandler, with distinct facial expressions, i.e., disgust and anger. previous methods use the face sequence containing both persons' faces as visual features, which will inevitably have a negative impact on predicting chandler's emotion. therefore, to fully leverage the visual modality for emotion recognition, it is crucial to extract the face sequence of the real speaker of each utterance. to this end, we propose a two-stage multimodal multi-task learning framework named facialmmt for the mermc task. in the first stage, we design a pipeline solution to obtain the face sequence of the real speaker, which contains three steps: 1) extract the face sequence containing all possible speakers based on the combination of multimodal rules and an active speaker detection model @xcite ; 2) identify the number of face clusters in the face sequence based on an unsupervised clustering algorithm named infomap @xcite ; 3) perform face matching and choose the face sequence with the highest confidence as the face sequence of the real speaker. based on the extracted face sequence, in the second stage, we further propose a multimodal facial expression-aware multi-task learning model named mario. mario first resorts to an auxiliary framelevel facial expression recognition task to obtain the emotion distribution of each frame in the face sequence. next, the emotion-aware visual representation is then integrated with textual and acoustic representations via cross-modal transformer @xcite for utterance-level emotion recognition. our main contributions can be summarized as follows: ## conclusion in this paper, we proposed a two-stage framework named facial expression-aware multimodal multi-task learning (facialmmt) for the mermc task. facialmmt first extracts the real speaker's face sequence from the video, and then leverages an auxiliary frame-level facial expression recognition task to obtain the emotion-aware visual representation through multi-task learning, followed by multimodal fusion for the mermc task. experiments on the meld dataset show the effectiveness of facialmmt.