Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection (2020)

Abstract: Sarcasm is a pervasive phenomenon in today’s social media platforms such as Twitter and Reddit. These platforms allow users to create multi-modal messages, including texts, images, and videos. Existing multi-modal sarcasm detection methods either simply concatenate the features from multi modalities or fuse the multi modalities information in a designed manner. However, they ignore the incongruity character in sarcastic utterance, which is often manifested between modalities or within modalities. Inspired by this, we propose a BERT architecture-based model, which concentrates on both intra and inter-modality incongruity for multi-modal sarcasm detection. To be specific, we are inspired by the idea of self-attention mechanism and design inter-modality attention to capturing inter-modality incongruity. In addition, the co-attention mechanism is applied to model the contradiction within the text. The incongruity information is then used for prediction. The experimental results demonstrate that our model achieves state-of-the-art performance on a public multi-modal sarcasm detection dataset.

## introduction sarcasm is a form of figurative language where the literal meaning of words does not hold, and instead, the opposite interpretation is intended @xcite . sarcasm is prevalent in today's social media platforms, and it can completely flip the polarity of sentiment or opinion. thus, an effective sarcasm detector is beneficial to applications like sentiment analysis, opinion mining @xcite , and other tasks that require people's real sentiment. however, the figurative nature of sarcasm makes it a challenging task @xcite . the scholars notice that sarcasm is often associated with a concept called incongruity which is used to suggest a distinction between reality and expectation (gibbs jr et @xcite . consequently, many approaches for sarcasm detection have been proposed by capturing the incongruity within text @xcite @xcite . more and more applications like twitter allow users to post multi-modal messages. accordingly, only modeling the incongruity within text modality is not enough to identify the inter-modality contradiction's sarcasm. consider the given examples in figure 1 ; people can not recognize sarcasm merely from text unless they find the contradiction between text and images. as a result, capturing the incongruity between modalities is significant for multi-modal sarcasm detection. however, the existing models for multi-modal sarcasm detection either concatenate the features from multi modalities @xcite or fuse the information from different modalities in a designed manner @xcite . previous multimodal sarcasm detection approaches neglect the incongruity character of sarcasm. we believe that it is meaningful to capture both intra and inter-modality incongruity for multi-modal sarcasm detection. we treat images and text as two modalities in this work and propose a novel bert architecture-based model for multi-modal sarcasm detection. bert as a pre-trained language model proposed by @xcite , which can be used to produce outstanding representations of text. for this reason, we utilize bert to acquire the representation of text and the hashtags (use the word with a '#' in front to indicate the topic of the tweet) within the text. we notice that hashtags might contain the information that contrasts the text. @xcite also studies the sentiment and sarcasm with the help of hashtags. consequently, we apply a co-attention matrix to model the incongruity between text and hashtags as the intra-modality incongruity. besides, the self-attention mechanism considers the interaction between keys and queries and the inter-modality incongruity information can also be treated as an interaction between text and images. as a result, inspired by the key idea of self-attention, we design the inter-modality attention which treats textual features as queries, image features as keys and values to capture the intermodality incongruity. the intra and inter-modality incongruity information are then combined for prediction. the main contributions of our work can be summarised as follows: ## method in this section, we first define the multi-modal sarcasm detection task. we then briefly present the background of the bert model and describe the architecture of our proposed model in detail. figure 2 gives an overview of our model. ## experiment this section first describes the dataset, experimental settings, baseline models, and experimental results. then, we conduct a series of ablative experiments to verify the components' effectiveness in training development testing sentences 19816 2410 2409 positive 8642 959 959 negative 11174 1451 1450 avg length 15.71 15.72 15.89 ## conclusion in this paper, we propose a novel bert architecture-based model to address the issue that existing multi-modal sarcasm detection approaches do not consider incongruity character of sarcasm. to be specific, our model considers both intra and inter-modality incongruity and achieves state-ofthe-art performance on a public multi-modal sarcasm detection dataset. besides, we also conduct a series of experiments to verify the effectiveness of our model. finally, we perform error analysis and find that the text on the images is essential for multi-modal sarcasm detection.