Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association (2020)

Abstract: Sarcasm is a sophisticated linguistic phenomenon to express the opposite of what one really means. With the rapid growth of social media, multimodal sarcastic tweets are widely posted on various social platforms. In multimodal context, sarcasm is no longer a pure linguistic phenomenon, and due to the nature of social media short text, the opposite is more often manifested via cross-modality expressions. Thus traditional text-based methods are insufficient to detect multimodal sarcasm. To reason with multimodal sarcastic tweets, in this paper, we propose a novel method for modeling cross-modality contrast in the associated context. Our method models both cross-modality contrast and semantic association by constructing the Decomposition and Relation Network (namely D&R Net). The decomposition network represents the commonality and discrepancy between image and text, and the relation network models the semantic association in cross-modality context. Experimental results on a public dataset demonstrate the effectiveness of our model in multimodal sarcasm detection.

## introduction sarcasm is a sophisticated linguistic phenomenon, defined by merriam-webster dictionary as 'the use of words that mean the opposite of what you really want to say, especially in order to insult someone, to show irritation, or to be funny'. it can not only disguise the hostility of the speaker, but also enhance the effect of mockery or humor on the listener @xcite . as an important clue to analyze people's true sentiment and intentions in communication from implicit expressions, automatic sarcasm detection plays a significant role in various applications that require the knowledge of people's sentiment or opinion @xcite , such as customer service, political stance detection, and user intent recognition. existing work on sarcasm detection mainly focuses on text data. early feature engineering approaches rely on the signal indicators of sarcasm, such as syntactic patterns, lexical indicators and special symbols @xcite @xcite . as sarcasm is often associated with implicit contrast or disparity between conveyed sentiment and user's situation in context @xcite , contextual contrast information at conversation, tweet or word level is also employed to detect sarcasm in text @xcite @xcite . recently, deep learning based methods are adopted to train end-to-end neural networks @xcite , achieving state-of-the-art performance. with the fast growing and diverse trend of social media, multimodal sarcastic tweets which convey abundant user sentiment are widely posted on various social platforms. there is a great demand for multimodal sarcasm detection to facilitate various applications. however, traditional text-based methods are not applicable to detect multimodal sarcastic tweets (fig. 1 ). in multimodal context, sarcasm is no longer a pure linguistic phenomenon, but rather the combined expressions of multiple modalities (i.e. text, image, etc.). as the short text in tweet often has insufficient contextual information, contextual contrast implied in multimodal sarcasm is typically conveyed by cross-modality expressions. for example, in fig. 1b , we can not reason about sarcasm intention simply from the short text 'perfect flying weather in april' until we notice the downpour outside the airplane window in the attached image. therefore, compared to text-based methods, the essential research issue in multimodal sarcasm detection is the reasoning of cross-modality contrast in the associated situation. several related work on multimodal sarcasm detection has been proposed @xcite @xcite . however, they mainly focus on the fusion of multimodal data, and did not address the above key research issue in reasoning with multimodal sarcasm. there are still two main research challenges for multimodal sarcasm detection. first, since sarcasm commonly manifests with a contrastive theme, this requires the detection model to have the ability to reason about cross-modality contrast or incongruity of situations. second, to ensure cross-modality contrast assessed in the associated common ground, the detection model needs the mechanism to concentrate on the semantic associated aspects of situations in cross-modality context. this contextual contrast and semantic association information acquired, in turn, can provide salient evidence to interpret the detection of multimodal sarcasm. to tackle the above challenges, in this paper, we propose a novel method to model both crossmodality contrast and semantic association by constructing the decomposition and relation network (i.e. d&r net) for multimodal sarcasm detection task. the decomposition network implicitly models cross-modality contrast information via representing the commonality and discrepancy between image and text in tweets. the relation network explicitly captures the semantic association between image and text via a cross-modality attention mechanism. the main contributions of our work are as follows: ## proposed model fig. 2 illustrates the overall architecture of our proposed d&r net for multimodal sarcasm detection, which is composed of four modules, preprocessing, encoding, decomposition network and relation network. we first preprocess the image and text inputs and extract adjective-noun pairs (anps) from each image. we then encode these triple inputs into hidden representations. after that, we learn to represent the commonality and discrepancy between image and text in decomposition network as well as the multi-view semantic association information in relation network. finally, we feed these crossmodality representations into classification module for multimodal sarcasm detection. ## conclusion in this paper, we identify the essential research issue in multimodal sarcasm detection. to model the cross-modality contrast in the associated context of multimodal sarcastic tweets, we propose the d&r net to represent the commonality and discrepancy between image and text and multi-view semantic associations in cross-modality context. our model is capable of reasoning multimodal sarcastic tweets with word-level interpretation. experimental results on a public dataset show that our model achieves the state-of-the-art performance compared with the existing models.