When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues (2022)

Abstract: Indirect speech such as sarcasm achieves a constellation of discourse goals in human communication. While the indirectness of figurative language warrants speakers to achieve certain pragmatic goals, it is challenging for AI agents to comprehend such idiosyncrasies of human communication. Though sarcasm identification has been a well-explored topic in dialogue analysis, for conversational systems to truly grasp a conversation’s innate meaning and generate appropriate responses, simply detecting sarcasm is not enough; it is vital to explain its underlying sarcastic connotation to capture its true essence. In this work, we study the discourse structure of sarcastic conversations and propose a novel task – Sarcasm Explanation in Dialogue (SED). Set in a multimodal and code-mixed setting, the task aims to generate natural language explanations of satirical conversations. To this end, we curate WITS, a new dataset to support our task. We propose MAF (Modality Aware Fusion), a multimodal context-aware attention and global information fusion module to capture multimodality and use it to benchmark WITS. The proposed attention module surpasses the traditional multimodal fusion baselines and reports the best performance on almost all metrics. Lastly, we carry out detailed analysis both quantitatively and qualitatively.

## introduction the use of figurative language serves many communicative purposes and is a regular feature of both oral and written communication @xcite . predominantly used to induce humour, criticism, or mockery @xcite , paradoxical language is also used in concurrence with hyperbole to show surprise @xcite as well as highlight the disparity between expectations and reality @xcite . while the use and comprehension of sarcasm is a * equal contribution given a sarcastic dialogue, the aim is to generate a natural language explanation for the sarcasm in it. blue text represents the english translation for the text. for dialogue agents, understanding sarcasm is even more crucial as there is a need to normalize its sarcastic undertone and deliver appropriate responses. conversations interspersed with sarcastic statements often use contrastive language to convey the opposite of what is being said. in a real-world setting, understanding sarcasm goes beyond negat-ing a dialogue's language and involves the acute comprehension of audio-visual cues. additionally, due to the presence of essential temporal, contextual, and speaker-dependent information, sarcasm understanding in conversation manifests as a challenging problem. consequently, many studies in the domain of dialogue systems have investigated sarcasm from textual, multimodal, and conversational standpoints @xcite @xcite . however, baring some exceptions @xcite @xcite , research on figurative language has focused predominantly on its identification rather than its comprehension and normalization. this paper addresses this gap by attempting to generate natural language explanations of satirical dialogues. to illustrate the proposed problem statement, we show an example in figure 1 . it contains a dyadic conversation of four utterances u 1 , u 2 , u 3 , u 4 , where the last utterance (u 4 ) is a sarcastic remark. note that in this example, although the opposite of what is being said is, "i don't have to think about it," it is not what the speaker means; thus, it enforces our hypothesis that sarcasm explanation goes beyond simply negating the dialogue's language. the discourse is also accompanied by ancillary audio-visual markers of satire such as an ironical intonation of the pitch, a blank face, or roll of the eyes. thus, conglomerating the conversation history, multimodal signals, and speaker information, sed aims to generate a coherent and cohesive natural language explanation associated with sarcastic dialogues. for the task at hand, we extend masac @xcite -a sarcasm detection dataset for codemixed conversations -by augmenting it with natural language explanations for each sarcastic dialogue. we name the dataset wits 1 . the dataset is a compilation of sarcastic dialogues from a popular indian tv show. along with the textual transcripts of the conversations, the dataset also contains multimodal signals of audio and video. we experiment with unimodal as well as multimodal models to benchmark wits. text, being the driving force of the explanations, is given the primary importance, and thus, we compare a number of established text-based sequence-to-sequence systems on wits. to incorporate multimodal information, we propose a unique fusion scheme of 1 wits: "why is this sarcastic" multimodal context-aware attention (mca2). inspired by @xcite , this attention variant facilitates deep semantic interaction between the multimodal signals and textual representations by conditioning the key and value vectors with audio-visual information and then performing dot product attention with these modified vectors. the generated audio and video information-informed textual representations are then combined using the global information fusion mechanism (gif). the gating mechanism of gif allows for the selective inclusion of information relevant to the satirical language and also prohibits any multimodal noise from seeping into the model. we further propose maf (modality aware fusion) module where the aforementioned mechanisms are introduced in the generative pretrained models (gplms) as adapter modules. our fusion strategy outperforms the textbased baselines and the traditional multimodal fusion schemes in terms of multiple text-generation metrics. finally, we conduct a comprehensive quantitative and qualitative analysis of the generated explanations. in a nutshell, our contributions are four fold: • we propose sarcasm explanation in dialogue (sed), a novel task aimed at generating a natural language explanation for a given sarcastic dialogue, elucidating the intended irony. reproducibility: the source codes and the dataset can be found here: https://github.com/lcs2-iiitd/maf.git . ## related work sarcasm and text: @xcite presented a well-compiled survey on computational sarcasm where the authors expanded on the relevant datasets, trends, and issues for automatic sarcasm identification. early work in sarcasm detection dealt with standalone text inputs like tweets and reviews @xcite @xcite . these initial works mostly focused on the use of linguistic and lexical features to spot the markers of sarcasm @xcite . more recently, attention-based architectures are proposed to harness the inter-and intra-sentence relationships in texts for efficient sarcasm identification @xcite @xcite . analysis of figurative language has also been extensively explored in conversational ai setting. @xcite utilised attentionbased rnns to identify sarcasm in the presence of context. two separate lstms-with-attention were trained for the two inputs (sentence and context) and their hidden representations were combined during the prediction. the study of sarcasm identification has also expanded beyond the english language. bharti et al. (2017) collected a hindi corpus of 2000 sarcastic tweets and employed rule-based approaches to detect sarcasm. swami et al. (2018) curated a dataset of 5000 satirical hindi-english code-mixed tweets and used n-gram feature vectors with various ml models for sarcasm detection. other notable studies include arabic (abu farha and magdy, 2020), spanish (ortega-bueno et al., 2019), and italian (cignarella et al., 2018) languages. ## dataset situational comedies, or 'sitcoms', vividly depict human behaviour and mannerism in everyday reallife settings. consequently, the nlp research community has successfully used such data for sarcasm identification @xcite . however, as there is no current dataset tailored for the proposed task, we curate a new dataset named wits, where we augment the already existing masac dataset @xcite with explanations for our task. masac is a multimodal, multi-party, hindi-english code-mixed dialogue dataset compiled from the popular indian tv show, 'sarabhai v/s sarabhai' foot_0 . we manually analyze the data and clean it for our task. while the original dataset contained 45 episodes of the tv series, we add 10 more episodes along with their transcription and audio-visual boundaries. subsequently, we select the sarcastic utterances from this augmented dataset and manually define the utterances to be included in the dialogue context for each of them. finally, we are left with 2240 sarcastic dialogues with the number of contextual utterances ranging from 2 to 27. each of these instances is manually 5 10 15 20 25 number of utterances in a dialog 0 100 200 300 400 500 number of dialogs (a) utterance length distribution 1 2 3 4 5 6 number of speakers in a dialog 0 200 400 600 800 1000 1200 1400 number of dialogs (b) speaker distribution m ay a in d ra va rd h an sa h il m o n is h a ro se sh o th er s source 0 100 200 300 400 500 600 700 target maya indravardhan sahil monisha rosesh others (c) source-target pair distribution ## proposed methodology in this section, we present our model and its nuances. the primary goal is to smoothly integrate multimodal knowledge into the bart architecture. to this end, we introduce multimodal aware fusion (maf), an adapter-based module that comprises of multimodal context-aware attention (mca2) and global information fusion (gif) mechanisms. positional encodings bart encoder output linear linear linear key query value acoustic context aware self-attention acoustic context visual context aware self-attention visual input transformer linear acoustic compound gate visual compound gate x x pointwise addition ## experiments, results and analysis in this section, we illustrate our feature extraction strategy, the comparative systems, followed by the results and its analysis. for a quantitative analysis of the generated explanations, we use the standard metrics for generative tasks -rouge-1/2/l @xcite , bleu-1/2/3/4 @xcite , and meteor @xcite . to capture the semantic similarity, we use the multilingual version of the bertscore @xcite . ## conclusion in this work, we proposed the new task of sarcasm explanation in dialogue (sed), which aims to generate a natural language explanation for sarcastic conversations. we curated wits, a novel multimodal, multiparty, code-mixed, dialogue dataset to support the sed task. we experimented with multiple text and multimodal baselines, which give promising results on the task at hand. furthermore, we designed a unique multimodal fusion scheme to merge the textual, acoustic, and visual features via multimodal context-aware attention (mca2) and global information fusion (gif) mechanisms. as hypothesized, the results show that acoustic and visual features support our task and thus, generate better explanations. we show extensive qualitative analysis of the explanations obtained from different models and highlight their advantages as well as pitfalls. we also perform a thorough human evaluation to compare the performance of the models with that of human understanding. though the models augmented with the proposed fusion strategy perform better than the rest, the human evaluation suggested there is still room for improvement which can be further explored in future studies.