Multimodal Sarcasm Target Identification in Tweets (2022)

Abstract: Sarcasm is important to sentiment analysis on social media. Sarcasm Target Identification (STI) deserves further study to understand sarcasm in depth. However, text lacking context or missing sarcasm target makes target identification very difficult. In this paper, we introduce multimodality to STI and present Multimodal Sarcasm Target Identification (MSTI) task. We propose a novel multi-scale cross-modality model that can simultaneously perform textual target labeling and visual target detection. In the model, we extract multi-scale visual features to enrich spatial information for different sized visual sarcasm targets. We design a set of convolution networks to unify multi-scale visual features with textual features for cross-modal attention learning, and correspondingly a set of transposed convolution networks to restore multi-scale visual information. The results show that visual clues can improve the performance of TSTI by a large margin, and VSTI achieves good accuracy.

## introduction sarcasm is a type of sentiment in which people express their negative feelings using positive or intensified positive words. it has the power to disguise the hostility of the speaker @xcite , thereby enhancing the effect of mockery or humor on the listener. sarcasm is prevalent on today's social media platforms such as twitter, and automatic sarcasm target identification (sti) bears great significance in customer service, opinion mining, and online harassment detection. previous works about sti have focused on text modality and proposed some methods such as rulebased, statistical classifier-based @xcite , and deep learning models with socio-linguistic features @xcite . however, detecting a sarcasm target with only text modality is not sufficient and complete. for example, in figure 1 (a), we are not sure whether the context conveys positive or negative emotions if we only see the text "this guy definitely deserves $15 an hour!". however, the negative information comes from the image. when observing a lazy guy in the picture lying on the chair, we can easily determine that the lazy guy is a sarcasm target and label the text "this guy" as a sarcasm target (st). moreover, the sarcasm target sometimes does not appear explicitly in the text, which is marked as 'outside'. in alta shared task (molla and @xcite , 'outside' cases account for over 30% of the data. for example, in the tweet of @xcite , the author teased that the skirt was too long, similar to a bed sheet; therefore, the sarcasm target should be the long skirt. however, no sarcasm target appears in the text but we can label the long skirt as an st with a blue bounding box in the picture. the above examples illustrate the necessity of combining images for sti. in this paper, we introduce a novel task called multimodal sarcasm target identification (msti) on social media data. the msti task is to extract sarcasm targets (sts) from both texts and images in tweets. the textual st is a word or a phrase, and the visual st is an object labeled by a bounding box, as shown in figure 1 . the challenge of the msti task is not only to extract both textual and visual features but also to leverage cross-modality interaction and semantic learning to improve the performance of sti. the contributions of this paper can be summarized as follows: ## related work the existing research on sarcasm analysis mainly focuses on sarcasm detection (sd) and sarcasm target identification (sti). we begin the literature review with textual sarcasm. ## the proposed approach in this section, we introduce a novel neural architecture for msti, as shown in figure 2 . ## conclusions in this paper, we introduce a new task for identifying both textual and visual sarcasm targets. this work provides a good attempt to detect sarcasm targets on images. our model integrates multiple components such as sequence labeling, multi-scale cross-modality learning, and object detection. the experimental results not only illustrate that visual clues can improve the performance of tsti by a large margin, approximately 5% in f1 score, but also prove that it is feasible to detect sarcasm targets in images, obtaining a good accuracy of 51.9% in ap 50 .