Continuous Attentive Multimodal Prompt Tuning for Few-Shot Multimodal Sarcasm Detection (2024)

Abstract: With the steep rise in multimodal content on social media, multimodal sarcasm detection has gained widespread attention from research communities. Existing studies depend on large-scale data, which is challenging to obtain and expensive to annotate. Thus, investigating this problem in a few-shot scenario is required. Overtly complex multimodal models are prone to overfitting on in-domain data, which hampers their performance on out-of-distribution (OOD) data. To address these issues, we propose Continuous Attentive Multimodal Prompt Tuning model (CAMP), that leverages the prompt tuning paradigm to handle few-shot multimodal sarcasm detection. To overcome the siloed learning process of continuous prompt tokens, we design a novel, continuous multimodal attentive prompt where the continuous tokens intricately engage with both image and text tokens, enabling the assimilation of knowledge from different input modalities. Experimental results indicate that our method outperforms other multimodal baseline methods in the few-shot setting and OOD scenarios.

## introduction sarcasm is a figurative language where the utterance conveys a meaning opposite to the literal meaning of the words used. detecting sarcasm is important for effectively understanding sentiment @xcite , hate speech @xcite , and users' opinions on social media @xcite @xcite . with the rise in multimodal content on social media platforms, multimodal sarcasm detection has gained widespread attention from research communities. multiple modalities provide a crucial clue to ascertain the sarcastic nature of a post since deciphering sarcasm from uni-modal (only text or image) content may be highly ambiguous or unspecified. current approaches for multimodal image-text sarcasm detection @xcite @xcite @xcite @xcite suffer from some major challenges. these models primarily rely on large annotated datasets to achieve good performance. however, these datasets are difficult to obtain, and annotation is expensive and highly challenging due to socio-cultural and contextual dependencies @xcite @xcite . distant supervision techniques for labeling, like the use of special markers such as #sarcasm on twitter, introduce additional noise in the form of wrong labels @xcite . due to the complex structure of these multimodal models, they tend to overfit on in-domain data causing a reduction in performance for out-ofdistribution (ood) data. prompt-based methods have gained popularity in few-shot learning as they enable pretrained language models (plms) to generalize to new tasks with minimal or no training data as plms can serve as knowledge bases @xcite due to their large-scale training on huge corpora. hence, it is imperative to use promptbased method for our task. most of the existing prompt-based works on downstream tasks are based on prompt-based finetuning @xcite @xcite , where discrete prompts are given as inputs to plms, and the entire plm is finetuned to fill up the mask token. this poses three main challenges. first, finding the right prompt in the discrete token space is difficult and often yields sub-optimal performance. changes in token count drastically impact results @xcite . second, training all model weights increases parameters, memory use, and training time. lastly, fine-tuning pre-trained language models often leads to catastrophic forgetting @xcite , reducing generalizability and performance on out-of-distribution data due to changes in the pre-trained weights. motivated by the shortcomings of traditional multimodal approaches and discrete prompt-based techniques, we explore the idea of prompt tuning @xcite , a new paradigm involving plms where task-specific continuous prompts are learned during training, keeping the parameters of the plm frozen. further, in the vanilla prompt tuning approach, a significant limitation arises from the frozen nature of the pretrained language model (plm). this constraint results in independent learning of continuous prompt tokens without integrating knowledge on how to attend to both image and text tokens effectively. to address this challenge, we propose a novel model: camp (continuous attentive multimodal prompt tuning) model for few-shot multimodal sarcasm detection. to begin with, we design multimodal continuous prompts with text and image modalities. we also use captions of the images as the third modality to bridge the semantic gap between image and text. our approach enhances the model's ability to better learn the continuous prompt tokens by incorporating multimodal information and introducing attentive mechanisms, thereby significantly improving its capacity to attend to both image and text tokens seamlessly. our results show that using only 0.3 fraction of the entire plm parameters, camp can achieve state-of-the-art results in few-shot multimodal sarcasm detection. camp also shows strong performance on the ood setting. in summary, the main contributions and findings of this paper are listed below: ## ablation experiments with our ablation experiments, we try to answer the following research questions. (1) is continuous attentive multimodal prompt better than its non-attentive counterpart? (2) how effective are continuous prompt tokens over their discrete counterparts for multimodal sarcasm detection? (3) do captions reduce the semantic gap between image and text modalities? ## conclusion in this paper, we tackled the problem of few-shot multimodal sarcasm detection. unlike traditional approaches that rely on early or late image-text fusion to learn the subtle interaction between the image and text modalities, we reformulate the problem as a cloze-filling task. to this end, we propose a novel approach of using continuous attentive multimodal prompt for this task. these attentive, continuous prompt tokens can effectively attend to the image and text modalities tokens and can dynamically adapt according to the input instance. our extensive experiments over two datasets demonstrate the effectiveness of our model, which outperforms strong baselines in few-shot and out-of-distribution (ood) settings. we also demonstrate the efficacy of our model camp over other discrete token-based techniques, including prompting and prompt-based finetuning, through several ablation experiments.