Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement (2022)

Abstract: Sarcasm is a linguistic phenomenon indicating a discrepancy between literal meanings and implied intentions. Due to its sophisticated nature, it is usually difficult to be detected from the text itself. As a result, multi-modal sarcasm detection has received more and more attention in both academia and industries. However, most existing techniques only modeled the atomic-level inconsistencies between the text input and its accompanying image, ignoring more complex compositions for both modalities. Moreover, they neglected the rich information contained in external knowledge, e.g., image captions. In this paper, we propose a novel hierarchical framework for sarcasm detection by exploring both the atomic-level congruity based on multi-head cross attentions and the composition-level congruity based on graph neural networks, where a post with low congruity can be identified as sarcasm. In addition, we exploit the effect of various knowledge resources for sarcasm detection. Evaluation results on a public multi-modal sarcasm detection dataset based on Twitter demonstrate the superiority of our proposed model.

## introduction sarcasm refers to satire or ironical statements where the literal meaning of words is contrary to the authentic intention of the speaker to insult someone or humorously criticize something. sarcasm detection has received considerable critical attention because sarcasm utterances are ubiquitous in today's social media platforms like twitter and reddit. however, it is a challenging task to distinguish sarcastic posts to date in light of their highly figurative nature and intricate linguistic synonymy @xcite . early sarcasm detection methods mainly relied on fixed textual patterns, e.g., lexical indicators, syntactic rules, specific hashtag labels and emoji occurrences @xcite ; maynard and figure 1 : an example of sarcasm along with the corresponding image and different types of external knowledge extracted from the image. the sarcasm sentence represents the need for some good news. however, the image of the tv program is switched to bad news depicting severe storms (bad weather) which contradicts the sentence. @xcite , which usually had poor performances and generalization abilities by failing to exploit contextual information. to resolve this issue, @xcite @xcite considered sarcasm contexts or the sentiments of sarcasm makers as useful clues to model congruity level within texts to gain consistent improvement. however, purely text-modality-based sarcasm detection methods may fail to discriminate certain sarcastic utterances as shown in figure 1 . in this case, it is hard to identify the actual sentiment of the text in the absence of the image forecasting severe weather. as text-image pairs are commonly observed in the current social platform, multi-modal methods become more effective for sarcasm prediction by capturing congruity information between textual and visual modalities @xcite @xcite @xcite . however, most of the existing multi-modal techniques only considered the congruity level between each token and image-patch @xcite and ignored the importance of multigranularity (e.g., granularity such as objects, and relations between objects) alignments, which have been proved to be effective in other related tasks, such as cross-modal retrieval @xcite and image-sentence matching @xcite . in fact, the hierarchical structures of both texts and images advocate for compositionlevel modeling besides single tokens or image patches @xcite . by exploring compositional semantics for sarcasm detection, it helps to identify more complex inconsistencies, e.g., inconsistency between a pair of related entities and a group of image patches. moreover, as figurativeness and subtlety inherent in sarcasm utterances may bring a negative impact to sarcasm detection, some works @xcite found that the identification of sarcasm also relies on the external knowledge of the world beyond the input texts and images as new contextual information. what's more, it has drawn increasing research interest in how to incorporate knowledge to boost many machine learning algorithms such as recommendation system @xcite and relation extraction @xcite . indeed, several studies extracted image attributes @xcite or adjective-noun pairs (anps) @xcite from images as visual semantic information to bridge the gap between texts and images. however, constrained by limited training data, such external knowledge may not be sufficient or accurate to represent the images (as shown in figure 1 ) which may bring negative effects for sarcasm detection. therefore, how to choose and leverage external knowledge for sarcasm detection is also worth being investigated. to tackle the limitations mentioned above, in this work, we propose a novel hierarchical framework for sarcasm detection. specifically, our proposed method takes both atomic-level congruity between independent image objects and tokens, as well as composition-level congruity considering object relations and semantic dependencies to promote multi-modal sarcasm identification. to obtain atomic-level congruity, we first adopt the multihead cross attention mechanism @xcite to project features from different modalities into the same space and then compute a similarity score for each token-object pair via inner products. next, we obtain composition-level congruity based on the output features of both textual modality and visual modality acquired in the previous step. concretely, we construct textual graphs and visual graphs using semantic dependencies among words and spatial dependencies among regions of objects, respectively, to capture composition-level feature for each modality using graph attention networks @xcite . our model concatenates both atomic-level and composition-level congruity features where semantic mismatches between the texts and images in different levels are jointly considered. specially, we elaborate the terminology used in our paper again: congruity represents the semantic consistency between image and text. if the meaning of the image and text pair is contradictory, this pair will get less congruity. atomic is between token and image patch, and compositional is between a group of tokens (phrase) and a group of patches (visual object). last but not the least, we propose to adopt the pre-trained transferable foundation models (e.g., clip @xcite ) to extract text information from the visual modality as external knowledge to assist sarcasm detection. the rationality of applying transferable foundation models is due to their effectiveness on a comprehensive set of tasks (e.g., descriptive and objective caption generation task) based on the zero-shot setting. as such, the extracted text contains ample information of the image which can be used to construct additional discriminative features for sarcasm detection. similar to the original textual input, the generated external knowledge also contains hierarchical information for sarcasm detection which can be consistently incorporated into our proposed framework to compute multi-granularity congruity against the original text input. the main contributions of this paper are summarized as follows: 1) to the best of our knowledge, we are the first to exploit hierarchical semantic interactions between textual and visual modalities to jointly model the atomic-level and compositionlevel congruities for sarcasm detection; 2) we propose a novel kind of external knowledge for sarcasm detection by using the pre-trained foundation model to generate image captions which can be naturally adopted as the input of our proposed framework; 3) we conduct extensive experiments on a publicly available multi-modal sarcasm detection benchmark dataset showing the superiority of our method over state-of-the-art methods with additional improvement using external knowledge. ## methodology our proposed framework contains four main components: feature extraction, atomic-level cross-modal congruity, composition-level cross-modal congruity and knowledge enhancement. given an input text-image pair, the feature extraction module aims to generate text features and image features via a pre-trained text encoder and an image encoder, respectively. these features will then be fed as input to the atomic-level cross-modal congruity module to obtain congruity scores via a multi-head cross attention model (mca). to produce composition-level congruity scores, we construct a textual graph and a visual graph and adopt graph attention networks (gat) to exploit complex compositions of different tokens as well as image objects. the input features to the gat are taken from the output of the atomic-level module. due to the page limitation, we place our illustration figure in figure 6 . our model is flexible to incorporate external knowledge as a "virtual" modality, which could be used to generate complementary features analogous to the image modality for congruity score computation. ## conclusion in this paper, we propose to tackle the problem of sarcasm detection by reasoning atomic-level congruity and composition-level congruity in a hierarchical manner. specifically, we propose to model the atomic-level congruity based on the multi-head cross attention mechanism and the compositionlevel congruity based on graph attention networks. in addition, we propose to exploit the effect of various knowledge resources on enhancing the discriminative power of the model. evaluation results demonstrate the superiority of our proposed model and the benefit of image captions as external knowledge for sarcasm detection.