Multi-modal Sarcasm Generation: Dataset and Solution (2023)

Abstract: As an interesting and challenging task, sarcasm generation has attracted widespread attention. Although very recent studies have made promising progress, none of them considers generating a sarcastic description for a given image - as what people are doing on Twitter. In this paper, we present a Multi-modal Sarcasm Generation (MSG) task: Given an image with hashtags that provide the sarcastic target, MSG aims to generate sarcastic descriptions like humans. Different from textual sarcasm generation, MSG is more challenging as it is difficult to accurately capture the key information from images, hashtags, and OCR tokens and exploit multi-modal incongruity to generate sarcastic descriptions. To support the research on MSG, we develop MuSG, a new dataset with 5000 images and related Twitter text. We also propose a multi-modal Transformer-based method as a solution to this MSG task. The input features are embedded in the common space and passed through the multi-modal Transformer layers to generate the sarcastic descriptions by the auto-regressive paradigm. Both automatic and manual evaluations demonstrate the superiority of our method. The dataset and code will be available soon.

## introduction sarcasm is a type of emotional expression that indirectly expresses contempt, shows irritation, or demonstrates humor. as a typical task on sarcasm, sarcasm generation (sg) is proposed to generate a sarcastic message for a given literal input @xcite , which can express a variety of communicative intent such as evoking humor and diminishing or enhancing critique @xcite . it can impact many downstream applications such as personalized dialog systems @xcite *: corresponding author and news comment generation @xcite . since sg is proposed, a surge of follow-up studies have been conducted @xcite @xcite @xcite . notably, the aforementioned sg studies have only been investigated in the textual field so far. however, nowadays social platforms usually leverage multi-modal data where visual information is integrated with the text, making the analysis of uni-modal data in isolation and limitation. therefore, research on multi-modal sarcasm is crucial and imperative. studies on multi-modal sarcasm can be categorized into threefold: multi-modal sarcasm detection @xcite @xcite @xcite , multimodal target identification @xcite , and multi-modal sarcasm explanation @xcite . unfortunately, there has been no research touching sarcasm generation facing multi-modal information until now. we hope general artificial intelligence to learn creativity and associative skills, so learning to generate sarcastic descriptions towards multi-modal inputs like humans deserves to study deeply. therefore, we propose a multi-modal sarcasm generation task (msg), which aims to generate sarcastic descriptions on social platforms like humans for a given image with the help of hashtags (cf. figure 1 ). compared with textual sarcasm generation, msg is more challenging. firstly, human expressions on social platforms are usually stylized with too many verbalized expressions like abbreviations and interjections. secondly, accurately capturing information from the key visual regions which may contribute to the sarcasm remains a question. finally, the incongruity between images and generated sarcastic descriptions reflects human creativity, imagination, and associative skills, which are hard for machines to learn and construct. to support the studies on msg, we develop musg, a new dataset consisting of 5000 images and related sarcastic descriptions. we manually collect samples with clear sarcastic target from twitter api and two existing multi-modal sarcasm detection datasets @xcite . the descriptions come with hashtags (the tokens with a '#' to indicate the topic of twitter) that point the way to sarcasm generation (cf. figure 1 #bankholidayweekend). the images contain ocr tokens information that can provide an associative context for sarcasm generation (cf. figure 1 did somebody say sunday funday). with the well-formed dataset musg, researchers can easily conduct studies on msg. consequently, as shown in figure 1 , we design a multi-modal transformer-based model (mtmsg) as a strong baseline for the proposed msg task. concretely, we first model spatial, semantic, and visual reasoning relations between multiple ocr tokens, hashtags, and visual features. further, we map all the modality-specific features to the same reference and utilize the self-attention mechanism @xcite to capture the relationships between them. finally, we combine the vocabulary with ocr tokens which usually contain sarcastic intent to capture the incongruity for sarcasm generations. with the ability to capture the intra-and inter-modality incongruity, our model is thus capable of effectively generating sarcastic descriptions. our contributions can be summarized as follows: ## dataset and metrics in this section, we describe how the new dataset is constructed and how performance is evaluated. ## methodology in this section, we describe our proposed mtmsg, a multi-modal transformer-based model for msg. the input to this task is an image and the hashtags of the corresponding twitter text, while the output is the generated descriptions that need to compare with the original twitter text (ground truth). yet, demonstrated by @xcite , the information of ocr text usually provides the context of sarcasm, which may contribute to sarcasm generation. therefore, we leverage the information from visual objects, hashtags, and ocr tokens for msg. the architecture of the proposed mtmsg is illustrated in figure 2 . specifically, we first embed the three modalities in the same reference, and then feed them into a multi-modal transformer to achieve intra-and inter-modality interactions. finally, our models learn to generate sarcastic descriptions through iterative decoding with the help of a dynamic pointer network. in the decoding process, we leverage the previous output to predict the next generated word in an auto-regressive manner. ## conclusion in this paper, we introduce a novel task of multimodal sarcasm generation, aiming to generate sarcastic descriptions like humans. to address the task, we investigate a new dataset, musg, containing 5000 images with corresponding sarcastic descriptions. further, we propose a strong baseline, mtmsg, to benchmark the musg dataset. machine evaluation metrics demonstrate that our proposed mtmsg outperforms various comparison baselines. moreover, the human evaluation shows that our proposed msg task is challenging and worth further in-depth research. we consider that msg opens a new avenue in the domains of sarcasm understanding and generation. in the future, we will explore the detection of key information from the images and the understanding of the intent from the ocr tokens.