C ofi P ara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models (2024)

Abstract: Social media abounds with multimodal sarcasm, and identifying sarcasm targets is particularly challenging due to the implicit incongruity not directly evident in the text and image modalities. Current methods for Multimodal Sarcasm Target Identification (MSTI) predominantly focus on superficial indicators in an end-to-end manner, overlooking the nuanced understanding of multimodal sarcasm conveyed through both the text and image. This paper proposes a versatile MSTI framework with a coarse-to-fine paradigm, by augmenting sarcasm explainability with reasoning and pre-training knowledge. Inspired by the powerful capacity of Large Multimodal Models (LMMs) on multimodal reasoning, we first engage LMMs to generate competing rationales for coarser-grained pre-training of a small language model on multimodal sarcasm detection. We then propose fine-tuning the model for finer-grained sarcasm target identification. Our framework is thus empowered to adeptly unveil the intricate targets within multimodal sarcasm and mitigate the negative impact posed by potential noise inherently in LMMs. Experimental results demonstrate that our model far outperforms state-of-the-art MSTI methods, and markedly exhibits explainability in deciphering sarcasm as well.

## introduction sarcasm, a prevalent form of figurative language, is often used in daily communication to convey irony, typically implying the opposite of its literal meaning @xcite . as an important component in deciphering sarcasm, automated sarcasm target identification (sti) is crucial for natural language processing (nlp) in customer service @xcite , opinion mining @xcite , and online harassment detection @xcite . although prior research on sti has primarily centered on textual content @xcite , the surge in multimodal user-generated content has propelled the field of multimodal sarcasm target identification to the forefront of research @xcite , making it a significant area of study in both nlp applications and multimedia computing. the msti task is to extract the entities being ridiculed (i.e., sarcasm targets) from both the text and image in multimodal sarcastic content. previous work @xcite attempted to straightforwardly integrate a bert-based textual encoder and a cnn-based visual encoder for just modeling the sarcasm text and image, respectively. the state-of-the-art approach @xcite treats msti merely as an end-to-end task, primarily focusing on the superficial signals evident in the surface-level text and image. however, a more thorough investigation and understanding of the underlying meanings are essential, particularly in cases where the correlation between image and text is not immediately apparent in multimodal sarcasm @xcite . comprehending and analyzing multimodal sarcasm poses a considerable challenge, because its implicit meaning demands an in-depth understanding and reasoning of commonsense knowledge. for example, as shown in figure 1 (a), a human checker needs reasonable thoughts between visual and textual sarcasm targets, to understand that the man's leisurely sitting posture in front of the control panel creates a sarcastic contrast between the idea of the train driver's job being difficult and the actual scene. moreover, as the example shows in figure 1 (b), the sarcasm target sometimes does not appear explicitly in the text, which makes it more challenging for conventional models to cognize that the example implies that the presence of the pepsi bottle is a form of product placement, which is often seen as a marketing tactic. we contend the challenge lies in delivering rich multimodal knowledge that consistently assists in deciphering the concealed semantics within the multimodal nature of sarcasm. in this paper, we adhere to the following two key principles in the knowledge-augmented design of our approach: 1) lmm reasoning: to grasp the implicit meanings intrinsic in sarcasm, we resort to the extensive prior knowledge embedded within large multimodal models (lmms) @xcite . this design philosophy enables complex reasoning, thereby enhancing both the msti accuracy and explainability; 2) msd pre-training: previous literature @xcite indicates that msti inherently appraises the presence of sarcasm targeting each entity within sarcastic content. similar to multimodal sarcasm detection (msd) @xcite , which involves determining the sarcasm in multi-modalities at a holistic content level, msti actually engages in finer-grained sarcasm detection at the localized entity level. considering such a close correlation between msd and msti, it is assumed that insights from the coarser-grained msd are instrumental in discerning sarcasm targets in the finer-grained msti. thus we devise a cohesive framework to operate on the coarse-to-fine training paradigm, aimed at pinpointing nuanced visual and textual targets of multimodal sarcasm for msti by benefitting from lmm reasoning and msd pre-training. to these ends, we propose a novel framework with a coarse-to-fine paradigm, cofipara, by leveraging the divergent knowledge extracted from lmms for multimodal sarcasm target identification. specifically, we integrate text and image modalities within the coarse-to-fine training paradigm, which consists of two phases: 1) coarser-grained sarcasm detection: initially, we engage lmms in critical thinking to generate rationales from both sarcastic and non-sarcastic perspectives. utilizing these generated sarcasm rationales, we pre-train a smaller model to act as a rationale referee to implicitly extract sarcasm-indicative signals in the competing rationales for sarcasm prediction. this process aligns multimodal features between the sarcasm content and its underlying rationales, alleviating the negative impact of inevitable noise from lmms through competing rationales; 2) finer-grained target identification: subsequently, we further fine-tune the smaller model pre-trained in the previous stage for multimodal sarcasm target identification. this phase enhances our model with the multimodal reasoning knowledge, acquired in the pre-training stage and the rationale in sarcastic perspective, to reveal the meanings concealed within the comprehensive multimodal information of sarcasm samples. in this manner, our cofipara framework could be naturally output as the explanatory basis for deciphering multimodal sarcasm. extensive experiments conducted on two public sarcasm datasets reveal that our approach far outperforms previous state-of-the-art msti methods, and achieves competitive results compared with msd baselines. the experimental analysis further underscores the enhanced ability to provide superior explainability in the realm of multimodal sarcasm. our contributions are summarized as follows in three folds: ## our approach problem statement. we define a multimodal sample as @xmath0, t }, which consists of an image i and a text t . in the context of the coarser-grained msd task, the label y of the sample falls into either of the categories: sarcastic or non-sarcastic. as for the finer-grained msti task, the label y is a tuple consisting of a textual sarcasm target y text , and a visual bounding box y img , where the model is tasked with pinpointing the sarcasm entity targeted within the provided text and image modalities of the sarcastic sample. in this paper, we focus on improving the finer-grained msti task by leveraging insights from the coarser-grained msd task. closed to the msd task, which establishes the presence of sarcasm in holistic semantics at the coarser level, the msti task inherently detects sarcasm targeting each entity of the multimodal sarcastic content to explicitly identify the specific sarcasm targets at the finer level @xcite . this work is designed mainly to integrate msd and msti into a versatile framework with the coarseto-fine training paradigm, which utilizes msd as the predecessor foundational stage to facilitate the lm decoder layer cross attention add ... ## conclusion and future work in this paper, we proposed a novel coarse-to-fine paradigm to decode the implicit meanings hidden beneath the surface of texts and images in multimodal sarcasm for msti, by leveraging rich prior knowledge from lmm reasoning and msd pretraining. we first inspired divergent thinking with lmms to derive competing rationales for coarsergrained pre-training of a small language model on msd. then we conducted finer-grained finetuning of the model on msti. comprehensive experiments and analyses confirm the advantages of our framework. future efforts aim to enhance our research by focusing on explicitly extracting useful information from generated rationales, to further relieve the inherent bias and variation in lmms.