Towards Exploiting Sticker for Multimodal Sentiment Analysis in Social Media: A New Dataset and Baseline (2022)

Abstract: Sentiment analysis in social media is challenging since posts are short of context. As a popular way to express emotion on social media, stickers related to these posts can supplement missing sentiments and help identify sentiments precisely. However, research about stickers has not been investigated further. To this end, we present a Chinese sticker-based multimodal dataset for the sentiment analysis task (CSMSA). Compared with previous real-world photo-based multimodal datasets, the CSMSA dataset focuses on stickers, conveying more vivid and moving emotions. The sticker-based multimodal sentiment analysis task is challenging in three aspects: inherent multimodality of stickers, significant inter-series variations between stickers, and complex multimodal sentiment fusion. We propose SAMSAM to address the above three challenges. Our model introduces a flexible masked self-attention mechanism to allow the dynamic interaction between post texts and stickers. The experimental results indicate that our model performs best compared with other models. More researches need to be devoted to this field. The dataset is publicly available at https://github.com/Logos23333/CSMSA .

## introduction in recent years, social media has become more and more popular, and people tend to express opinions and emotions on social media @xcite . collecting opinions and analyzing sentiment in social media can help a lot for marketing @xcite , campaign trends prediction, and so forth. the sentiment analysis is challenging in social media because of the lack of context @xcite . stickers attached to posts can supplement missing sentiment and help to identify sentiment precisely. such task of performing sentiment analysis with multiple data sources is called multimodal sentiment analysis. figure 1 : an example of online conversation. the first image is a photo that does not reflect any sentiment. the second is a sticker, expressing the emotional tendency of happiness. the sticker here contributes to supplementing the missing sentiment of the text. many research efforts have been devoted to multimodal sentiment analysis @xcite . however, most of the multimodal sentiment analysis datasets are based on real-world photos @xcite . research focusing on sticker-based multimodal sentiment is limited. in social media, people tend to express the sentiment with stickers rather than real-world photo @xcite . stickers can convey more vivid and direct emotion. for example, as shown in figure 1 , the first image is a photo related to the post context, but it does not reflect any sentiment. instead, the second image, i.e., the sticker, shows a sense of happiness, revealing the missing sentiment of the corresponding texts. to this end, we introduce csmsa, a challenging chinese sticker based multimodal dataset for sentiment analysis task in social media. csmsa includes 28k text-sticker pairs, including 1.5k sentiment-annotated pairs, and 16k different stickers. this dataset is the first annotated stickerbased dataset for multimodal sentiment analysis to the best of our knowledge. our dataset will release to encourage research on multimodal sentiment analysis in social media. the proposed csmsa task is challenging in three aspects: first, stickers may be inherently multimodal because they are embedded with texts foot_0 , while other datasets have only real-world photos. for example, as shown in figure 2 , sticker (a) is attached with the text "i'm beat" and sticker (b) is attached with the text "i'm so touched". the difficulty is that the same sticker with different sticker texts may vary significantly in sentiment. second, stickers are highly different in styles, leading models to learn robust representations for the stickers following various distributions hardly @xcite . in contrast, most of the photo-based datasets only consist of human faces or food @xcite , and the styles do not differ much. considering the large inter-series variation of stickers, the csmsa task requires models to adapt to different artistic styles. third, the sentiment fusion of text and stickers is complex. for example, the text sentiment in figure 2 is negative obviously. however, after combining the text and the sticker (b), the multimodal sentiment is positive, showing a sense of moving and touching. multimodal sentiment does not always favor a single modality, making the fusion between modalities complex. in general, the first and second challenges are due to the difficulty of modeling the stickers, making it more challenging to fuse the sentiment with the post text. we propose a sticker-aware multimodal sentiment analysis model (samsam) to address the above three challenges. to address the first and second challenges, samsam introduces the sticker text and sticker series tag, which can be seen as a further complement to sticker sentiment to model the sticker wholly and accurately. to address the third challenge, samsam adopts the flexible masked attention mechanism to allow post texts and stickers to interact fully but selectively. the masked attention mechanism can help the model extract the most helpful information for the current sentiment judgment. we conduct experiments on the proposed csmsa dataset. the experimental results show that our model performs best in the challenging csmsa task. the ablation study indicates that the encoding of sticker texts and sticker series help in understanding multimodal sentiment, and the proposed masked self-attention mechanism can improve model performance significantly. the contributions of this paper are as follows: ## conclusion in this paper, we propose to focus on stickers rather than the real-world photo in the field of multimodal sentiment analysis. the sentiment fusion of stickers and texts is complex and challenging. compared with real-world photos, the sticker is inherently multimodal, and it has a significant inter-series variation, making it difficult to encode. we propose a sticker-based dataset for the sentiment analysis task, with 1.5k text-sticker human-annotated pairs. this dataset is the first annotated sticker-based dataset for multimodal sentiment analysis to the best of our knowledge. experimental results validate the challenges of the csmsa task. previous models cannot be directly applied to this task. more researches need to be devoted to this field.