Multimodal End-to-End Sparse Model for Emotion Recognition (2021)

Abstract: Existing works in multimodal affective computing tasks, such as emotion recognition and personality recognition, generally adopt a two-phase pipeline by first extracting feature representations for each single modality with hand crafted algorithms, and then performing end-to-end learning with extracted features. However, the extracted features are fixed and cannot be further fine-tuned on different target tasks, and manually finding feature extracting algorithms does not generalize or scale well to different tasks, which can lead to sub-optimal performance. In this paper, we develop a fully end-to-end model that connects the two phases and optimizes them jointly. In addition, we restructure the current datasets to enable the fully end-to-end training. Furthermore, to reduce the computational overhead brought by the end-to-end model, we introduce a sparse cross-modal attention mechanism for the feature extraction. Experimental results show that our fully end-to-end model significantly surpasses the current state-of-the-art models based on the two-phase pipeline. Moreover, by adding the sparse cross-modal attention, our model can maintain the performance with around half less computation in the feature extraction part of the model.

## introduction humans show their characteristics through not only the words they use, but also the way they speak and their facial expressions. therefore, in multimodal affective computing tasks, such as emotion recognition, there are usually three modalities: textual, acoustic, and visual. one of the main challenges in these tasks is how to model the interactions between different modalities, as they contain both supplementary and complementary information @xcite . ## related works human affect recognition is a popular and widely studied research topic @xcite @xcite . in recent years, there is a trend to leverage multimodal information to tackle these research tasks, such as emotion recognition @xcite , sentiment analysis @xcite , personality trait recognition @xcite , etc, have drawn more and more attention. different methods have been proposed to improve the performance and crossmodal interactions. in earlier works, early fusion @xcite and late fusion @xcite of modalities were widely adopted. later, more complex approaches were proposed. for example, @xcite introduced the tensor fusion network to model the interactions of the three modalities by performing the cartesian product, while @xcite used an attention gate to shift the words using the visual and acoustic features. in addition, based on the transformer @xcite , @xcite introduced the multimodal transformer to improve the performance given unaligned multimodal data, and @xcite introduced a multimodal adaptation gate to integrate visual and acoustic information into a large pre-trained language model. however, unlike some other multimodal tasks @xcite @xcite using fully end-to-end learning, all of these methods require a feature extraction phase using hand-crafted algorithms (details in section 5.2), which makes the whole approach a twophase pipeline. ## dataset reorganization the fully end-to-end multimodal model requires the inputs to be raw data for the three modalities (visual, textual and acoustic). the existing multimodal emotion recognition datasets cannot be directly applied for the fully end-to-end training for two main reasons. first, the datasets provide split of training, validation and test data for the hand-crafted features as the input of the model and emotion or sentiment labels as the output of the model. however, this dataset split cannot be directly mapped to the raw data since the split indices cannot be matched back to the raw data. second, the labels of the data samples are aligned with the text modality. however, the visual and acoustic modalities are not aligned with the textual modality in the raw data, which disables the fully end-to-end training. to make the existing datasets usable for the fully end-to-end training and evaluation, we need to reorganize them according to two steps: 1) align the text, visual and acoustic modalities; 2) split the aligned data into training, validation and test sets. in this work we reorganize two emotion recognition datasets: interactive emotional dyadic motion capture (iemocap) and cmu multimodal opinion sentiment and emotion intensity (cmu-mosei). both have multi-class and multi-labelled data for multimodal emotion recognition obtained by generating raw utterance-level data, aligning the three modalities, and creating a new split over the aligned data. in the following section, we will first introduce the existing datasets, and then we will give a detailed description of how we reorganize them. ## ablation study we conduct a comprehensive ablation study to further investigate how the models perform when one or more modalities are absent. the results are shown in ## conclusion and future work in this paper, we first compare and contrast the two-phase pipeline and the fully end-to-end (fe2e) modelling of the multimodal emotion recognition task. then, we propose our novel multimodal end-to-end sparse model (mesm) to reduce the computational overhead brought by the fully endto-end model. additionally, we reorganize two existing datasets to enable fully end-to-end training. the empirical results demonstrate that the fe2e model has an advantage in feature learning and surpasses the current state-of-the-art models that are based on the two-phase pipeline. furthermore, mesm is able to halve the amount of computation in the feature extraction part compared to fe2e, while maintaining its performance. in our case study, we provide a visualization of the crossmodal attention maps on both visual and acoustic data. it shows that our method can be interpretable, and the cross-modal attention can successfully select important feature points based on different emotion categories. for future work, we believe that incorporating more modalities into the sparse cross-modal attention mechanism is worth exploring since it could potentially enhance the robustness of the sparsity (selection of features).