Multi-Modal Sarcasm Detection via Cross-Modal Graph Convolutional Network (2022)

Abstract: With the increasing popularity of posting multimodal messages online, many recent studies have been carried out utilizing both textual and visual information for multi-modal sarcasm detection. In this paper, we investigate multi-modal sarcasm detection from a novel perspective by constructing a cross-modal graph for each instance to explicitly draw the ironic relations between textual and visual modalities. Specifically, we first detect the objects paired with descriptions of the image modality, enabling the learning of important visual information. Then, the descriptions of the objects are served as a bridge to determine the importance of the association between the objects of image modality and the contextual words of text modality, so as to build a cross-modal graph for each multi-modal instance. Furthermore, we devise a cross-modal graph convolutional network to make sense of the incongruity relations between modalities for multi-modal sarcasm detection. Extensive experimental results and in-depth analysis show that our model achieves state-of-the-art performance in multi-modal sarcasm detection.

## introduction sarcasm is a peculiar form of sentiment expressions, allowing individuals to express contempt sentiment or intention that is converse to the authentic/apparent sentiment information @xcite @xcite . as such, accurately detecting satirical/ironic expression could potentially improve the performance of sentiment analysis and opinion mining @xcite @xcite . in today's fast growing social media platforms, it is common to post multi-modal messages. therefore, in addition to developing sarcasm detection models for textual data @xcite , it is increasingly popular to explore sarcasm detection in multi-modal data such as text and images @xcite . dealing with multimodal data requires an understanding of the information presented in different modalities. as the sarcastic example shown in figure 1 (a ), text-only approaches may erroneously identify it as a positive sentiment expression due to the phrase "wonderful weather". this post however contains a sarcastic expression with negative sentiment, because it is accompanied by an image with "thunderstorm clouds". the key of effective multi-modal sarcasm detection is to accurately extract the incongruent sentiment cues from different modalities, allowing the detection of the true sentiment conveyed in the message. to perform multi-modal sarcasm detection on data composed of text and image, several related research efforts attempt to concatenate the textual and visual features to fuse sarcastic information @xcite , employ attention mechanism to implicitly fuse the features of different modalities based on external knowledge @xcite @xcite , or build interactive graphs to model the relations of different modalities @xcite . despite promising progress made by existing models, they still suffer from the following limitations: 1) simply considering the whole image does not produce good results, mostly due to the intricate visual information presented in an image; not to mention that only particular visual patches are related to the text. as in the examples shown in figure 1 , the correct results can be easily obtained by only tracking the visual information in the bounding boxes. therefore, discriminating key visual objects from the irrelevant ones could lead to improved learning of visual information. 2) crucial visual information that relates to the sarcastic cues of text modality may be scattered in an image (figure 1 (b) ). as such, it is essential to focus on drawing the intricate sentiment connections between text and image modalities, allowing a good exploitation of the contradictory sentiment information between modalities for learning sarcastic clues. to this end, we propose a novel cross-modal graph convolutional networks (cmgcn) by constructing a cross-modal graph for each instance, where the important visual information and the related textual tokens are explicitly linked. this allows for the extraction of incongruous implications between two modalities in sarcasm detection. concretely, instead of trying to produce a caption of the whole image, we first detect the objects of the image to capture the important visual regions and the corresponding attribute-object pairs via the approach proposed by @xcite . then, we explore a novel solution to assign weights to the edges of the cross-modal graph by means of computing the word similarities between the object descriptors of the attribute-object pairs and textual words based on the wordnet @xcite . further, to introduce the multi-modal sentiment relations into the cross-modal graphs, inspired by @xcite , we devise a modulat-ing factor of sentiment relation for each edge by retrieving the affective weights of attribute descriptors (usually adjectives with affective information) and textual words from external affective knowledge (senticnet @xcite ). as such, the modulating factors can be adopted to refine the edge weights of word similarities, allowing the capture of sentiment incongruities of the cross-modal nodes in the graph. further, in the light of crossmodal graphs, we deploy a gcn architecture to make sense of the incongruous relations across the modalities for multi-modal sarcasm detection. the main contributions of our work are summarized as follows: ## methodology in this section, we describe our proposed cross-modal graph convolutional networks (cmgcn) model for multi-modal sarcasm detection in details. as demonstrated in figure 2 , the architecture of the proposed cmgcn contains four main components: ## conclusion and future work this paper has proposed a novel cross-modal graph architecture for multi-modal sarcasm detection, in which the crucial visual regions can be explicitly connected to the highly correlated textual tokens for learning the incongruity sentiment of sarcastic expression. specifically, unlike previous research efforts that simply consider the visual information of the whole image, we attempt to recognize the important visual regions via object detection results, and further devise a novel cross-modal graph to explicitly establish the connections of scattered visual regions and the associated textual tokens. more concretely, owing to the object detection results, the attribute-object pair descriptors of the objects are served as a bridge to track the highly related sarcastic cues between image and text modalities and their connection weights, and then deriving the cross-modal graphs based on external knowledge bases. afterwards, a gcns architecture based on a retrieval-based attention mechanism is employed to capture the key incongruity sentiment expressions across different modalities for multi-modal sarcasm detection. to the best of our knowledge, it is the first study of utilizing a cross-modal graph to extract intricate multi-modal sarcastic relations via object detection and sentiment cues from external knowledge bases. extensive experiments on a public benchmark dataset show that our proposed approach significantly outperforms state-of-the-art baseline methods. as described in section 3.3, the weights of edges in the cross-modal graph are computed based on both word similarities and affective clues between textual words and the attribute-object pairs of the image regions, and the dependency tree of the textmodality. the approach can be easily generalized to other sentiment-related multi-modal learning scenarios. nevertheless, the cross-graph solution might not be generalized well to other multi-modal tasks or data genres, if there is a lack of affective knowledge or a difficulty in deriving depen-dency trees in low-resource settings. therefore, future research can consider exploiting alternatively approaches to automatically learn the weights of edges in the cross-modal graph without relying on external knowledge sources.