M - SENA : An Integrated Platform for Multimodal Sentiment Analysis (2022)

Abstract: M-SENA is an open-sourced platform for Multimodal Sentiment Analysis. It aims to facilitate advanced research by providing flexible toolkits, reliable benchmarks, and intuitive demonstrations. The platform features a fully modular video sentiment analysis framework consisting of data management, feature extraction, model training, and result analysis modules. In this paper, we first illustrate the overall architecture of the M-SENA platform and then introduce features of the core modules. Reliable baseline results of different modality features and MSA benchmarks are also reported. Moreover, we use model evaluation and analysis tools provided by M-SENA to present intermediate representation visualization, on-the-fly instance test, and generalization ability test results. The source code of the platform is publicly available at https://github.com/thuiar/M-SENA .

## introduction multimodal sentiment analysis (msa) aims to judge the speaker's sentiment from video segments @xcite @xcite . it has attracted increasing attention due to the booming of user-generated online content. although impressive improvements have been witnessed in recent msa researches @xcite @xcite , building an end-to-end video sentiment analysis system for real-world scenarios is still full of challenges. the first challenge lies in effective acoustic and visual feature extraction. most previous approaches @xcite @xcite are developed on the provided modality sequences from cmu-multimodalsdk 1 . however, reproducing exact identical acoustic and visual feature extraction is almost impossible due to the the vague description of feature selection and backbone selection (both covarep foot_0 and facet foot_1 can not be directly used in python). moreover, recent literature @xcite @xcite observe that the text modality stands in the predominant position while acoustic and visual modalities have few contributions to the final sentiment classification. such results further arouse the attention on effective feature extraction of acoustic and visual modalities. with the awareness of the importance of acoustic and visual feature extraction, researchers attempt to develop models based on customized modality sequences instead of provided features @xcite . however, performance comparison with different modality features is unfair. therefore, the demand for reliable comparison of modality features and fusion methods is increasingly urgent. another factor that limits the application of existing msa models in real scenarios is the lack of comprehensive model evaluation and analysis approaches. models obtained outstanding performance on the given test set might degrade in realworld scenarios due to the distribution discrepancy or random modality perturbations @xcite @xcite . besides, effective model analysis is also crucial for researchers to explain the improvements and perform model refinement. the multimodal sentiment analysis platform (m-sena) is developed to address the above challenges. for acoustic and visual features, the platform integrates librosa @xcite , opensmile @xcite , openface @xcite , mediapipe @xcite and provides a highly customized feature extraction api in python. with the modular msa pipeline, fair comparison between different features ## model analysis demonstration this section demonstrates model analysis results using the m-sena platform. intermediate result analysis is presented in section 4.1, on-the-fly instance analysis is shown in section 4.2, and generalization ability analysis is illustrated in section 4.3. ## related works to the best of our knowledge, there are two widely used open-source repositories from cmu team 6 and sutd team 7 . both of them provide tools to load well-known msa datasets and implement sev-eral benchmarks methods. so far, their works have attracted considerable attention and facilitated the birth of new msa models such as mult @xcite and mmim @xcite . in this paper, we propose m-sena, compared to previous works, the m-sena platform is novel from the following aspects. for data management, previous work directly loads the extracted features, while the m-sena platform focuses on intuitive raw video demonstration, and provides user with a convenient means for private dataset construction. for modality features, m-sena platform first provides user-customized feature extraction toolkit and a transparent feature extraction process. following the tutorial, users can easily reproduce the feature extraction steps and develop their research on designed feature set. for model training, the m-sena platform first utilizes a unified msa framework and provide an easy-to-reproduce model training api integrating fourteen msa benchmarks on three popular msa dataset. for model evaluation, the m-sena is the first msa platform consisting of comprehensive evaluation means stressing model robustness for real-world scenarios, which aims to bridge the gap between msa research and applications.