CH - SIMS : A C hinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality (2020)

Abstract: Previous studies in multimodal sentiment analysis have used limited datasets, which only contain unified multimodal annotations. However, the unified annotations do not always reflect the independent sentiment of single modalities and limit the model to capture the difference between modalities. In this paper, we introduce a Chinese single- and multi-modal sentiment analysis dataset, CH-SIMS, which contains 2,281 refined video segments in the wild with both multimodal and independent unimodal annotations. It allows researchers to study the interaction between modalities or use independent unimodal annotations for unimodal sentiment analysis. Furthermore, we propose a multi-task learning framework based on late fusion as the baseline. Extensive experiments on the CH-SIMS show that our methods achieve state-of-the-art performance and learn more distinctive unimodal representations. The full dataset and codes are available for use at https://github.com/thuiar/MMSA .

## introduction sentiment analysis is an important research area in natural language processing (nlp). it has wide applications for other nlp tasks, such as opinion mining, dialogue generation, and user behavior analysis. previous study @xcite mainly focused on text sentiment analysis and achieved impressive results. however, using text alone is not sufficient to determine the speaker's sentimental state, and text can be misleading. with the booming of short video applications, nonverbal behaviors (vision and audio) are introduced to solve the above shortcomings @xcite . in multimodal sentiment analysis, intra-modal representation and inter-modal fusion are two im-m: negative m: negative t : positive a : weakly positive v : negative (others) (ours) it is too unexpected figure 1 : an example of the annotation difference between ch-sims and other datasets. for each multimodal clip, in addition to multimodal annotations, our proposed dataset has independent unimodal annotations. m: multimodal, t: text, a: audio, v: vision. to validate the above analysis, in this paper, we propose a chinese multimodal sentiment analysis dataset with independent unimodal annotations, ch-sims. figure 1 shows an example of the annotation difference between our proposed dataset and the other existing multimodal datasets. sims has 2,281 refined video clips collected from different movies, tv serials, and variety shows with spontaneous expressions, various head poses, occlusions, and illuminations. the cheavd @xcite is also a chinese multimodal dataset, but it only contains two modalities (vision and audio) and one unified annotation. in contrast, sims has three modalities and unimodal annotations except for multimodal annotations for each clip. therefore, researchers can use sims to do both unimodal and multimodal sentiment analysis tasks. furthermore, researchers can develop new methods for multimodal sentiment analysis with these additional annotations. based on sims, we propose a multimodal multitask learning framework using unimodal and multimodal annotations. in this framework, the unimodal and multimodal tasks share the feature representation sub-network in the bottom. it is suitable for all multimodal models based on late-fusion. then, we introduce three late-fusion models, including tfn, lmf, and late-fusion dnn (lf-dnn), into our framework. with unimodal tasks, the performance of multimodal task is significantly increased. furthermore, we make a detailed discus-sion on multimodal sentiment analysis, unimodal sentiment analysis and multi-task learning. lastly, we verify that the introduction of unimodal annotations can effectively expand the difference between different modalities and obtain better performance in inter-modal fusion. in this work, we provide a new perspective for multimodal sentiment analysis. our main contributions in this paper can be summarized as follows: ## related work in this section, we briefly review related work in multimodal datasets, multimodal sentiment analysis, and multi-task learning. ## ch-sims dataset in this section, we introduce a novel chinese multimodal sentiment analysis dataset with independent unimodal annotations, ch-sims. in the following subsections, we will explain the data acquisition, annotation, and feature extraction in detail. ## multimodal multi-task learning framework in this section, we describe our proposed multimodal multi-task learning framework. shown as figure 3 , based on late-fusion multimodal learning framework @xcite , we add independent output units for three unimodal representations: text, audio, and vision. therefore, these unimodal representations not only participate in feature fusion but are used to generate their predictive outputs. for the convenience in following introduction, in text, audio and vision, we assume that l u , d u i , d u r , where @xmath0, a, v}, represent the sequence length, initial feature dimension extracted by section 3.3 and representation dimension learned by unimodal feature extractor, respectively. the batch size is b. ## experiments in this section, we mainly explore the following problems using sims: the aim is to show that the learned unimodal representations are more distinctive after using unimodal annotations. ## ablation study in this section, we compare the difference in the effects of combining different unimodal tasks on multimodal sentiment analysis. we aim to further explore the influence on multimodal sentiment analysis with different unimodal tasks. furthermore, we reveal the relationship between multi-task learning and multimodal sentiment analysis. we conducted multiple combination experiments to analyze the effects of different unimodal subtasks on the main multimodal task. in this part, we only report the results in mlf-dnn. results are shown in ## conclusion in this paper, we propose a novel chinese multimodal sentiment analysis dataset with independent unimodal annotations and a multimodal multi-task learning framework based on late-fusion methods. we hope that the introduction of ch-sims will provide a new perspective for researches on multimodal analysis. furthermore, we conduct extensive experiments on discussing unimodal, multimodal, and multi-task learning. lastly, we summarize our overall findings as follows: in the future, we will further explore the connection between multimodal analysis and multi-task learning and incorporate more fusion strategy, including early-and middle-fusion.