Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation (2023)

Abstract: Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which aims to generate a natural language sentence for a multimodal social post (an image as well as its caption) to explain why it contains sarcasm. Although the existing pioneer study has achieved great success with the BART backbone, it overlooks the gap between the visual feature space and the decoder semantic space, the object-level metadata of the image, as well as the potential external knowledge. To solve these limitations, in this work, we propose a novel mulTi-source sEmantic grAph-based Multimodal sarcasm explanation scheme, named TEAM. In particular, TEAM extracts the object-level semantic meta-data instead of the traditional global visual features from the input image. Meanwhile, TEAM resorts to ConceptNet to obtain the external related knowledge concepts for the input text and the extracted object meta-data. Thereafter, TEAM introduces a multi-source semantic graph that comprehensively characterize the multi-source (i.e., caption, object meta-data, external knowledge) semantic relations to facilitate the sarcasm reasoning. Extensive experiments on a public released dataset MORE verify the superiority of our model over cutting-edge methods.

## introduction sarcasm is a common linguistic phenomenon, especially in posts on online social media platforms, that expresses people's emotions or opinions in a contrary manner. since it benefits various realworld applications, such as customer feedback analysis and public opinion analysis, the sarcasm detection task has gained increasing research attention @xcite . despite related great studies of the task, they can only identify the sarcastic post but could not give the concrete explanation for why it is sarcastic, making their detection results less convincing. noticing this issue, recent studies have shifted to the task of sarcasm explanation, which aims to generate a natural language sentence to explain the intended irony in a sarcastic post. for example, peled and reichart utilized the recurrent neural network (rnn) @xcite -based encoder-decoder architecture to tackle the sarcasm interpretation task. although previous studies have attained impressive results, they focus on investigating the sarcasm explanation purely based on the textual input. nevertheless, with the advances of multimedia devices, people tend to express their emotions or opinions through multimodal social posts. moreover, the visual content usually also conveys important clues for explaining the sarcasm, as shown in figure 1 . motivated by this, desai et al. proposed the task of multimodal sarcasm explanation, which aims to generate the explanation for a multimodal input (i.e., an image plus its corresponding caption). the authors gave a solution that first fuses the multimodal features with a crossmodal attention module, and then generates the explanation with the decoder of bart, a popular generative pretrained language model. although this pioneer study has achieved promising performance, it still suffers from three key limitations. to tackle these limitations, we propose a novel multi-source semantic graph-based multimodal sarcasm explanation generation scheme, team for short, which explores three semantic sources: the input caption, object meta-data derived from the input image, as well as the external knowledge. specifically, team includes four components: vision-based object-level semantic extraction, external related knowledge acquisition, multisource semantic graph-based sarcasm reasoning, and sarcasm explanation generation. as shown in figure 2 , in the first module, we focus on extracting the semantic meta-data of the key objects in the input image instead of the conventional global visual features, to adapt the decoding space of bart and facilitate the fine-grained sarcasm reasoning. in the second module, we target at acquiring the external related knowledge concepts for the input caption and the extracted object meta-data, where a large-scale knowledge base conceptnet @xcite is used as the reference. in the third module, we construct the multi-source semantic graph to model the various semantic relations residing in the three semantic sources, and adopt gcn to fulfil the sarcasm reasoning. in the last module, we generate the target sarcasm explanation with the bart @xcite ) decoder based on the three semantic sources. we conduct extensive experiments on a public released multimodal sarcasm explanation dataset, on which our method outperforms the best baseline by 28.90 and 22.47 in terms of bleu-4 @xcite and rouge-l @xcite , respectively. our contributions can be concluded as follows. as a byproduct, we release our code and parameters foot_0 to facilitate this community. our work is related to sarcasm detection and sarcasm-related generation. ## task formulation suppose we have a training dataset d composed of n samples, i.e., for simplicity, we temporally omit the subscript i that indexes the training samples. ## conclusion and future work in this work, we propose a novel multi-source semantic graph-based multimodal sarcasm explanation generation scheme. experimental results on a public dataset demonstrate the superiority of our model over existing cutting-edge methods, and validate the advantage of utilizing the object-level meta-data over the global visual feature of the image as well as the benefit of incorporating the external knowledge in the context of multimodal sarcasm explanation. particularly, we notice that our model performs worse on ocr samples than on non-ocr samples. this is due to that our model currently ignores the text embedded in the image. in the future, we plan to incorporate the embedded text, which could indicate important clues for sarcasm explanation, to boost the model performance.