MMSD 2.0: Towards a Reliable Multi-modal Sarcasm Detection System (2023)

Abstract: Multi-modal sarcasm detection has attracted much recent attention. Nevertheless, the existing benchmark (MMSD) has some shortcomings that hinder the development of reliable multi-modal sarcasm detection system: (1) There are some spurious cues in MMSD, leading to the model bias learning; (2) The negative samples in MMSD are not always reasonable. To solve the aforementioned issues, we introduce MMSD2.0, a correction dataset that fixes the shortcomings of MMSD, by removing the spurious cues and re-annotating the unreasonable samples. Meanwhile, we present a novel framework called multi-view CLIP that is capable of leveraging multi-grained cues from multiple perspectives (i.e., text, image, and text-image interaction view) for multi-modal sarcasm detection. Extensive experiments show that MMSD2.0 is a valuable benchmark for building reliable multi-modal sarcasm detection systems and multi-view CLIP can significantly outperform the previous best baselines.

## introduction sarcasm detection is used to identify the real sentiment of user, which is beneficial for sentiment analysis and opinion mining task @xcite . recently, due to the rapid progress of social media platform, multi-modal sarcasm detection, which aims to recognize the sarcastic sentiment in multi-modal scenario (e.g., text and image modalities), has attracted increasing research attention. specifically, as illustrated in figure 1 , given the text-image pair, multi-modal sarcasm detection system predicts sarcasm label because the image shows a traffic jam that contradicts the text "love the traffic". unlike traditional sarcasm detection task, the uniqueness of multi-modal sarcasm detection lies in effectively modeling the consistency and sarcasm relationship among different modalities. thanks to the rapid development of deep neural networks, remarkable success has been witnessed in multi-modal sarcasm detection. specifically, @xcite makes the first attempt to explicitly concatenate the textual and visual features for multi-modal sarcasm detection. a series of works have employed attention mechanism to implicitly incorporate features from different modalities @xcite @xcite . more recently, graph-based approaches have emerged to identify significant cues in sarcasm detection @xcite @xcite , which is capable of better capturing relationship across different modalities, thereby dominating the performance in the literature. while current multi-modal sarcasm detection systems have achieved promising results, it is unclear whether these results faithfully reflect the multi-modal understanding ability of models. in fact, when a text-modality only model roberta is applied to multimodal sarcasm detection, its performance significantly surpasses the state-of-theart multi-modal model hke @xcite with 6.6% improvement (see detailed analysis § 4.3). such observation suggests that the performance of current models may heavily depend on spurious cues in textual data, rather than truly capturing the relationship among different modalities, resulting in low reliability. further exploration reveals that the characteristics of the mmsd benchmark @xcite ) may be the cause of this phenomenon: (1) spurious cues: mmsd benchmark has some spurious cues (e.g., hashtag and emoji word) occuring in an unbalance distribution of positive and negative examples, which leads to the model bias learning; (2) unreasonable annotation: mmsd benchmark simply assigns the text without special hashtag (e.g., #sarcasm, etc.) as negative examples (i.e., not sarcastic label). we argue that this construction operation is unreasonable because sentence without #sarcasm tag can also express the sarcastic intention. take utterance in figure 2 as an example, the utterance without #sarcasm tag still belong to the sarcastic sample. therefore, further chasing performance on current mmsd benchmark may hinder the development of reliable multi-modal sarcasm detection system. motivated by the above observation, we shift our eyes from traditional complex network design work @xcite @xcite to the establishment of reasonable multi-modal sarcasm detection benchmark. specifically, we introduce mmsd2.0 to address these problems. to solve the first drawback, mmsd2.0 removes the spurious cues (e.g., sarcasm word) from text in the mmsd, which encourages model to truly capture the relationship across different modalities rather than just memorize the spurious correlation. such operation can benefit the development of bias mitigation in multi-modal sarcasm detection studies. to address the second problem, we directly make our efforts to re-annotate the unreasonable data. specifically, given each utterance labeled with "not sarcastic" in the mmsd, we use crowdsourced workers to check and re-annotate the label. this correction process results in changes to over 50% of samples in original mmsd. in addition to the dataset contribution, we propose a novel framework called multi-view clip, which can naturally inherit multi-modal knowledge from pre-trained clip model. specifically, multi-view clip utilizes different sarcasm cues captured from multiple perspectives (i.e., text, image and text-image interaction view), and aggregates multi-view information for final sarcasm detection. compared to previous superior graphbased approaches, multi-view clip has the following advantages: (1) multi-view clip does not require any image pre-processing step for graph building (e.g., object detection); (2) multi-view clip does not require any complex network architecture and can naturally make full use of knowledge in vl pre-trained model for multi-modal sarcasm detection. contributions of this work can be summarized as follows: to facilitate future research, all datasets, code are publicly available at https://github.com/ joeying1019/mmsd2.0 . ## mmsd2.0 dataset construction mmsd2.0 aims to solve two main shortcomings of mmsd, which consists of (1) spurious cues removal ( §2.1) and (2) unreasonable samples re-annotation via crowdsourcing ( §2.2). this section provides mmsd2.0 dataset construction in detail and the overall process is illustrated in figure 2 . ## approach this section first describes the basic architecture for clip ( §3.1) and then illustrates the proposed multi-view clip framework ( §3.2). ## conclusion this paper first analyzed the underlying issues in current multi-modal sarcasm detection dataset and then introduced a mmsd2.0 benchmark, which takes the first meaningful step towards building a reliable multi-modal sarcasm detection system. furthermore, we proposed a novel framework multi-view clip to capture sarcasm cues from different perspectives, including image view, text view, and image-text interactions view. experimental results show that multi-view clip attains state-of-the-art performance.