Layer-wise Fusion with Modality Independence Modeling for Multi-modal Emotion Recognition (2023)

Abstract: Multi-modal emotion recognition has gained increasing attention in recent years due to its widespread applications and the advances in multi-modal learning approaches. However, previous studies primarily focus on developing models that exploit the unification of multiple modalities. In this paper, we propose that maintaining modality independence is beneficial for the model performance. According to this principle, we construct a dataset, and devise a multi-modal transformer model. The new dataset, CHinese Emotion Recognition dataset with Modality-wise Annotions, abbreviated as CHERMA, provides uni-modal labels for each individual modality, and multi-modal labels for all modalities jointly observed. The model consists of uni-modal transformer modules that learn representations for each modality, and a multi-modal transformer module that fuses all modalities. All the modules are supervised by their corresponding labels separately, and the forward information flow is uni-directionally from the uni-modal modules to the multi-modal module. The supervision strategy and the model architecture guarantee each individual modality learns its representation independently, and meanwhile the multi-modal module aggregates all information. Extensive empirical results demonstrate that our proposed scheme outperforms state-of-the-art alternatives, corroborating the importance of modality independence in multi-modal emotion recognition. The dataset and codes are availabel at https://github.com/sunjunaimer/LFMIM

## introduction the goal of human emotion recognition is to automatically detect or categorize the emotional states of human according to some inputs. nowadays, emotion recognition can be found in * corresponding author a broad range of applications, including but not limited to emotional support @xcite , human-computer interaction @xcite and healthcare surveillance @xcite . henceforth, emotion recognition has attracted increasing attention from both research community and industry in recent years @xcite . the early works perform emotion recognition primarily with a single modality @xcite @xcite , e.g., vision, text, audio and so on. recent multi-modal approaches have showcased more appealing performance than their uni-modal counterparts @xcite . however, most existing literature on multimodal learning overemphasizes the combination of different modalities without fully respecting modality independence, which might be harmful to the model. in the sequel, we illustrate this through the lens of datasets and model design. datasets current datasets for multi-modal emotion recognition are usually annotated with the joint observation of all modalities, resulting in shared labels for all modalities @xcite @xcite @xcite . this leads to the fact that all modalities in the multi-modal model are supervised by the same common labels, which reduces the modality diversity and might even mislead some modalities @xcite . in practice, it is anticipated that inconsistent labels will be attained if we annotate different modalities separately. in this circumstance, in order to learn diverse and modality-specific representations, the modules for different modalities are expected to be trained with their own labels rather than the common labels. ## related works there are a large volume of relevant works on multimodal emotion recognition, for which interested readers can refer to survey papers @xcite and references therein. in this section, we only cover the most related works, corresponding to the datasets and multimodal fusion models in the following. ## dataset description in this section, we give a detailed introduction to the new dataset-cherma. we will present how the data is collected and annotated, the characteristics of the data, and the pre-processing of the data for model training. before introducing the data, we give the definitions of some notations. let t, a, v represent the three modalities-text, audio, and vision, respectively; let m denote the joint of the three modalities. denote by x @xmath0, a, v}, the feature sequence of the corresponding modality, where t u and d u are the sequence length and the feature dimension, respectively. associated with each feature sequence is its unimodal labels and multi-modal label {y u |@xmath1, a, v, m}}. for our training dataset, we use ({x n u } @xmath2,a,v} , {y n u } @xmath3,a,v,m} ) for @xmath4, 2, • • • , n } to represent the n-th sample, where n denotes the total number of samples. in the rest of the paper, we sometimes drop the index n for brevity when no confusion occurs. ## experiments and analysis in this section, we first compare our proposed model with typical benchmark models to validate the effectiveness of our model. then we perform ablation studies to analyze the proposed model, and demonstrate the differences between our model and its compared counterparts. ## conclusions in this paper, we uphold modality independence for multi-modal emotion recognition in the context of modality inconsistency. therefore, we build a new dataset that includes uni-modal labels and multi-modal labels. our model maintains modality independence via 1) supervising each modality with its own labels, and 2) enforcing uni-directional information flow from uni-modal modules to multi-modal module. numerical results verify that independence indeed helps to gain more effective information from the modalities and improve the model performance for the multimodal emotion recognition. albeit independence benefits the multi-modal learning, it does not mean that individual modality should be prevented from exploring other modalities in any circumstance. there should be a sweet point between modality independence and dependence, which constitutes our future research interest.