R ethinking TMSC : An Empirical Study for Target-Oriented Multimodal Sentiment Classification (2023)

Abstract: Recently, Target-oriented Multimodal Sentiment Classification (TMSC) has gained significant attention among scholars. However, current multimodal models have reached a performance bottleneck. To investigate the causes of this problem, we perform extensive empirical evaluation and in-depth analysis of the datasets to answer the following questions: **Q1**: Are the modalities equally important for TMSC? **Q2**: Which multimodal fusion modules are more effective? **Q3**: Do existing datasets adequately support the research? Our experiments and analyses reveal that the current TMSC systems primarily rely on the textual modality, as most of targets’ sentiments can be determined *solely* by text. Consequently, we point out several directions to work on for the TMSC task in terms of model design and dataset construction. The code and data can be found in https://github.com/Junjie-Ye/RethinkingTMSC.

## introduction target-oriented sentiment classification, also known as aspect-based sentiment classification, is a fundamental task of sentiment analysis @xcite @xcite . it aims to judge the sentimental polarity (positive, negative, or neutral) of a specific target within text. to improve the performance by considering multimodal information, target-oriented multimodal sentiment classification (tmsc) is proposed to integrate both visual and textual information @xcite . recently, the performance of the tmsc systems gradually reaches a plateau and the progress in tackling this task has slowed down. using the f1-score metric on the popular datasets, twitter15 and twitter17 @xcite , we observe that state-of-the-art baselines only achieve ˚corresponding authors, jzhou@cs.ecnu.edu.cn, qz@fudan.edu.cn. an f1-score of around 70. therefore, in this paper, we aim to analyze the causes behind it at both model level and modality level. roughly speaking, the modules in the model structures can be categorized into two types: 1) encoders to model the representations of different modalities; and 2) multimodal fusion modules to model the interactions between modalities. moreover, we give a deep analysis of the characteristics of two widely-used datasets, aiming to answer the following three questions: q1: are the modalities equally important for tmsc? to explore this issue, we compare and analyze the performance of unimodal models on this task. for the textual modality, we use bert @xcite as the backbone, as it is a widely-used pre-trained language model outperforming earlier models like lstm @xcite , memory network @xcite , etc. for the visual modality, resnet @xcite , vit @xcite , and faster r-cnn (ren et al., 2015) are adopted (see figure 1 ). q2: which multimodal fusion modules are more effective? the current models use various fusion strategies to model the interactions between modalities, while obtaining little improvement. to explore the effectiveness of different fusion approaches, we summarize the fusion strategies into six categories: concatenation, tensor fusion @xcite , self attention, image2text, text2image and bi-direction. then we perform a comparative study of them using a unified setup to eliminate potential bias from model size and structure (see figure 2 ). q3: do existing datasets adequately support the research? we analyze the existing datasets (i.e., twitter15 and twitter17) in depth and obtain the following findings: 1) the size of existing datasets is limited; 2) the multimodal sentiment is much more consistent with the textual sentiment than the visual sentiment; 3) a large number of targets do not exist in images; 4) there are only a small number of samples where the sentiment is decided by both text and image. the main contributions of this work are as follows: 1) we investigate the effectiveness of different model structures for tmsc, including various unimodal encoders and multimodal fusion modules; 2) we give an in-depth analysis of limitations of existing widely-used datasets; 3) we derive several valuable observations and point out promising directions for the future research of tmsc model design and dataset creation. ## empirical study we summarize the model structures and performance of the baselines for the tmsc task in ## data analysis to gain a deeper understanding of the performance issues mentioned above, we conduct detailed analyses of the two datasets, taking into account quantity, diversity, and annotation. following the annotation procedure employed by @xcite , we enlist the participation of three domain experts to annotate 400 randomly sampled test data (200 from twitter15 and 200 from twitter17) across four aspects, with the majority vote being considered as the final annotation result (figure 3 ) 4 . we have the following observations: first, as shown in second, the multimodal sentiment has high consistency with the textual sentiment but low consistency with the visual sentiment. in twitter15, 4 illustrative examples with annotations are in appendix d. 93% of the targets have the same textual sentiment as the multimodal sentiment, while only 47.5% have a visual sentiment that matches. this indicates the biased distribution existing in the dataset, i.e., the textual information is more useful for determining the multimodal sentiment. although this phenomenon is mitigated in twitter17, the textual information is still more consistent with the multimodal sentiment than the visual information. third, a large number of targets do not exist in images, which is also not suitable for the targetoriented multimodal sentiment classification task. this phenomenon may stem from the construction of the two datasets, where the targets are selected directly from the text, without taking into account the corresponding images @xcite . fourth, due to the facts of irrelevant images and non-existence of targets in images, there is only a small portion of the data where the sentiment is determined by both text and images. specifically, only 22% of twitter15 and 55% of twitter17 data require both text and images for the sentiment classification. as for the multimodal task, these two datasets may not be the best-suited in this aspect. based on our analyses of existing datasets, we propose that high-quality tmsc datasets should possess the following characteristics: 1) accurately reflecting the real-world data distribution, including factors such as unbalanced label distribution, while also providing sufficient data samples for different cases; 2) large data diversity, i.e., various data types and domains, to facilitate valid testing for models' generalization capability and robustness; and 3) multi-dimensional annotation information, including both multimodal and unimodal sentiment, to enable thorough analysis of the model's ability to handle different data sources. ## conclusion and future work in this paper, we conduct a series of in-depth experiments for tmsc and data analysis of existing datasets. our findings reveal that current multimodal models do not exhibit significant performance gains compared to text-only models on the tmsc task. this is largely attributed to the over-reliance on textual modality in existing datasets, while visual modality playing a comparatively less significant role. based on our experimental analyses, we propose future directions for designing models for the tmsc task and for constructing more suitable datasets which better capture the multimodal nature of social media sentiments.