Leveraging Generative Large Language Models with Visual Instruction and Demonstration Retrieval for Multimodal Sarcasm Detection (2024)

Abstract: Multimodal sarcasm detection aims to identify sarcasm in the given image-text pairs and has wide applications in the multimodal domains. Previous works primarily design complex network structures to fuse the image-text modality features for classification. However, such complicated structures may risk overfitting on in-domain data, reducing the performance in out-of-distribution (OOD) scenarios. Additionally, existing methods typically do not fully utilize cross-modal features, limiting their performance on in-domain datasets. Therefore, to build a more reliable multimodal sarcasm detection model, we propose a generative multimodal sarcasm model consisting of a designed instruction template and a demonstration retrieval module based on the large language model. Moreover, to assess the generalization of current methods, we introduce an OOD test set, RedEval. Experimental results demonstrate that our method is effective and achieves state-of-the-art (SOTA) performance on the in-domain MMSD2.0 and OOD RedEval datasets.

## introduction sarcasm is a linguistic phenomenon of verbal irony where the literal meaning contradicts the real intent of the speaker. sarcasm detection aims to identify the actual sentiment of the user and can be widely applied in various scenarios such as public opinion mining @xcite and social media analysis @xcite . recently, due to the rapid surge of multimodal data on social media, multimodal sarcasm detection has gained increasing attraction and significance @xcite @xcite @xcite @xcite @xcite ., * corresponding author the left of the figure conveys a sarcastic meaning with the contrast between "grass replacing" and "wonderful job". the right part displays an image of "beautiful autumn", which is semantically consistent and shows no sarcasm. previous works rely on complex model structures for feature fusion followed by classification, whereas our method generates answers based on llms. fortunately, significant progress has been made in various nlp generation tasks with the development of large language models (llms) @xcite . further leveraging llms and extending them to multimodal domains, the multimodal large language models (mllms) @xcite @xcite have also significantly improved various multimodal tasks and show great generalization. therefore, to build a more reliable multimodal sarcasm detection model, we redefine multimodal sarcasm detection as a generative task to take advantage of the powerful mllms. to further leverage and enhance the performance of mllms, we design a detailed instruction template and propose a simple yet effective demonstration retrieval module. furthermore, considering the lack of research assessing the generalization of current multimodal sarcasm detection models, we collect multimodal data from other social media platforms and propose a new test set named redeval for ood evaluation. in summary, our contributions are as follows: ## methodology in this section, we present the overview of our method. we first present the brief task formulation and describe the mllm-based generative framework. then we detail our retrieval module and introduce the training and generation process. ## conclusion in this paper, we focus on the problems of insufficient ood generalization and inadequate utilization of cross-modal features in current multimodal sarcasm detection models. to build a more reliable model, we propose a generative multimodal sarcasm detection model consisting of an instruction template and a demonstration retrieval module based on the powerful multimodal large language model. moreover, to assess the generalization of current multimodal sarcasm detection models, we also propose a new ood test set, redeval. experimental results demonstrate that our method is effective and outperforms previous baselines by a large margin, achieving the sota performance on both in-domain mmsd2.0 and out-of-domain redeval datasets.