Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021)

Abstract: Multimodal fusion has been proved to improve emotion recognition performance in previous works. However, in real-world applications, we often encounter the problem of missing modality, and which modalities will be missing is uncertain. It makes the fixed multimodal fusion fail in such cases. In this work, we propose a unified model, Missing Modality Imagination Network (MMIN), to deal with the uncertain missing modality problem. MMIN learns robust joint multimodal representations, which can predict the representation of any missing modality given available modalities under different missing modality conditions. Comprehensive experiments on two benchmark datasets demonstrate that the unified MMIN model significantly improves emotion recognition performance under both uncertain missing-modality testing conditions and full-modality ideal testing condition. The code will be available at https://github.com/AIM3-RUC/MMIN .

## introduction automatic multimodal emotion recognition is very important to natural human-computer interactions @xcite . it aims to understand and interpret human emotions expressed through multiple modalities such as speech content, voice tones and facial expression. previous works have shown that these different modalities are complimentary for emotion expression, and proposed many effective multimodal fusion methods to improve the emotion recognition performance @xcite @xcite . however, in real applications, many common causes can lead to the missing modality problem. for example, the camera is turned off or blocked due to privacy issues; the speech content is unavailable due to automatic speech recognition errors; the voice and text are missing due to the silence of the user; or the faces cannot be detected due to lighting or occlusion issues as shown in figure 1 . existing multimodal fusion models trained on full-modality samples usually fail when partial modalities are missing @xcite @xcite . the missing modality problem has attracted more research attention in the past years, and the existing solutions for this problem are mainly based on learning joint multimodal representation so that all modality information can be encoded. han et al. @xcite propose a joint training approach that implicitly fuses multimodal information from auxiliary modalities, which improves the monomodal emotion recognition performance. the recent cross-modality sequential translation-based methods proposed in @xcite learn the joint multimodal representations via translating a source modality to multiple target modalities, which improves the performance of the source modality as input at the test time. however, these methods can only deal with the scenario where the source modality is input to the trained model. different models need to be built for different missing modality cases foot_0 . additionally, the sequential translation-based models require translation and generation of videos, audios, and text, which are difficult to train especially with limited training samples @xcite . in this work, we propose a novel unified model, missing modality imagination network (mmin), to address the above issues. specifically, the proposed mmin learns the robust joint multimodal representations through cross-modality imagination with cascade residual autoencoder (cra) @xcite and cycle consistency learning @xcite based on sentence-level modalityspecific representations, as the sentence-level representation is more reasonable for modeling the cross-modality emotion correlation. the imagination module aims to predict the sentence-level emotional representation of the missing modality from the other available modalities. to the best of our knowledge, this is the first work that investigates a unified model for multimodal emotion recognition with uncertain missing-modality. extensive experiments are carried out on two benchmark datasets, iemocap and msp-improv, under both uncertain missing-modality and full-modality conditions. the proposed mmin model as a unified multimodal emotion recognition model can learn robust joint multimodal representations and outperforms the standard multimodal fusion models on both benchmark datasets under both the uncertain missing-modality and the fullmodality conditions. furthermore, to evaluate the imagination ability of our mmin model, we visualize the distributions of the imagined representations of the missing modalities and its ground-truth representations and find they are very similar, which demonstrates that mmin can imagine the representations of the missing modalities based on the representations of the available modalities. in summary, the main contributions of this work are: 1) we propose a unified model, missing modality imagination network (mmin), to improve the robustness of emotion recognition systems under uncertain missing-modality testing con- data augmentation methods: parthasarathy et al. @xcite propose a strategy to randomly ablate visual inputs during training at the clip or frame level to mimic realworld missing modality scenarios for audio-visual multimodal emotion recognition, which improves the recognition performance under missing modality conditions. generative methods: tran et al. @xcite propose cascaded residual autoencoder (cra) to utilize the residual mechanism over the autoencoder structure, which can take the corrupted data and estimate a function to well restore the incomplete data. cai et al. @xcite propose an encoder-decoder deep neural network to generate the missing modality (positron emission tomography, pet) given the available modality (magnetic resonance imaging, mri), and the generated pet can provide complementary information to improve the detection and tracking of alzheimers disease. learning joint multimodal representations: han et al. @xcite propose a joint training model that consists of two modality-specific encoders and one shared classifier, which implicitly fuse the audio and visual information as joint representations and improve the performance of the mono-modality emotion recognition. pham et al. @xcite propose a sequential translation-based model to learn the joint representation between the source modality and multiple target modalities. the hidden vectors of the source modality encoder work as the joint representations, which improve the emotion recognition performance of the source modality. wang et al. @xcite follow this translation-based method and propose a more efficient transformerbased translation model with parallel translation including textual features to acoustic features and textual features to visual features. moreover, the above two translation-based models adopt the forward translation and backward translation training strategy to ensure that joint representations can retain maximal information from all modalities. ## method given a set of video segments s, we use @xmath0, x v , x t ) to represent the raw multimodal features for a video segment @xmath1, where x a , x v and x t represent the raw features of acoustic, visual and textual modalities respectively. |s| represents the number of video segments in set s. we denote the target set @xmath2, y @xmath3, 1, . . . , c}, where y i is the target emotion category of the video @xcite unified triplet format pairs 1 ((x a ), ( ## conclusion in this paper, we propose a novel unified multimodal emotion recognition model, missing modality imagination network (mmin), to improve the emotion recognition performance under uncertain missing-modality conditions in real application scenarios. the proposed mmin can learn the robust joint multimodal representations through cross-modality imagination via the cascade residual autoencoder and cycle consistency learning. extensive experiments on two public benchmark datasets demonstrate the effectiveness and robustness of our proposed model, which significantly outperforms other baselines under both uncertain missing-modality and full-modality conditions. in the future work, we will explore ways to further improve the robust joint multimodal representation.