Sentiment and Emotion help Sarcasm? A Multi-task Learning Framework for Multi-Modal Sarcasm, Sentiment and Emotion Analysis (2020)

Abstract: In this paper, we hypothesize that sarcasm is closely related to sentiment and emotion, and thereby propose a multi-task deep learning framework to solve all these three problems simultaneously in a multi-modal conversational scenario. We, at first, manually annotate the recently released multi-modal MUStARD sarcasm dataset with sentiment and emotion classes, both implicit and explicit. For multi-tasking, we propose two attention mechanisms, viz. Inter-segment Inter-modal Attention (Ie-Attention) and Intra-segment Inter-modal Attention (Ia-Attention). The main motivation of Ie-Attention is to learn the relationship between the different segments of the sentence across the modalities. In contrast, Ia-Attention focuses within the same segment of the sentence across the modalities. Finally, representations from both the attentions are concatenated and shared across the five classes (i.e., sarcasm, implicit sentiment, explicit sentiment, implicit emotion, explicit emotion) for multi-tasking. Experimental results on the extended version of the MUStARD dataset show the efficacy of our proposed approach for sarcasm detection over the existing state-of-the-art systems. The evaluation also shows that the proposed multi-task framework yields better performance for the primary task, i.e., sarcasm detection, with the help of two secondary tasks, emotion and sentiment analysis.

## introduction sarcasm is an essential aspect of daily conversation, and it adds more fun to the language. oscar wilde, an irish poet-playwright, quotes, "sarcasm is the lowest form of wit, but the highest form of intelligence". irrespective of its relation with intelligence, sarcasm is often challenging to understand. sarcasm is often used to convey thinly veiled disapproval humorously. this can be easily depicted through the following example, "this is so good, that i am gonna enjoy it in the balcony. i can enjoy my view, whilst i enjoy my dessert." this utterance, at an outer glance, conveys that the speaker is extremely pleased with his dessert and wants to elevate the experience by enjoying it in the balcony. but, careful observation of the sentiment and emotion of the speaker helps us understand that the speaker is disgusted with the dessert and has a negative sentiment during the utterance (c.f. figure 1 ). this is where sentiment and emotion come into the picture. sentiment, emotion and sarcasm are highly intertwined, and one helps in the understanding of the others better. even though sentiment, emotion, and sarcasm are related, sarcasm was treated separately from its other counterparts in the past due to its complexity and its high dependency on the context. moreover, multi-modal input helps the model to understand the intent and the sentiment of the speaker with more certainty. thus in the context of a dialogue, multi-modal data such as video @xcite along with text helps to understand the sentiment and emotion of the speaker, and in turn, helps to detect sarcasm in the conversation. in this paper, we exploit these relationships, and make use of sentiment and emotion of the speaker for predicting sarcasm, specifically for the task, in a multi-modal conversational context. the main contributions and/or attributes of our proposed research are as follows: (a). we propose a multi-task learning framework for multi-modal sarcasm, sentiment, and emotion analysis. we leverage the utility of sentiment and emotion of the speaker to predict sarcasm. in our multi-task framework, sarcasm is treated as the primary task, whereas emotion analysis and sentiment analysis are considered as the secondary tasks. @xcite . we also propose two attention mechanisms viz. i e -attention and i a -attention to better combine the information across the modalities to effectively classify sarcasm, sentiment, and emotion. (c). we annotate the recently released sarcasm dataset, mustard with sentiment and emotion classes (both implicit and explicit), and (d). we present the state-of-the-art for sarcasm prediction in multi-modal scenario. ## related work a survey of the literature suggests that a multimodal approach towards sarcasm detection is a fairly new approach rather than a text-based classification. traditionally, rule-based classification @xcite approaches were used for sarcasm detection. @xcite have exploited sentiment and emotion features extracted from the pre-trained models for sentiment, emotion, and personality on a text corpus, and use them to predict sarcasm through a convolutional neural network. in recent times, the use of multi-modal sources of information has gained significant attention to the researchers for affective computing. @xcite proposed a new two-level strategy (divide, conquer, and combine) for feature fusion through a hierarchical feature fusion network for multimodal affective computing. @xcite exploits the interaction between a pair of modalities through an application of inter-modal interaction module (iim) that closely follows the concepts of an auto-encoder for the multi-modal sentiment and emotion analysis. @xcite proposed a contextual inter-modal attention based framework for multi-modal sentiment classification. in other work @xcite , an attention-based multitask learning framework has been introduced for sentiment and emotion recognition. although multi-modal sources of information (e.g., audio, visual, along with text) offers more evidence in detecting sarcasm, this has not been attempted much, one of the main reasons being the non-availability of multi-modal datasets. recently, researchers @xcite have started exploiting multi-modal sources of information for sarcasm detection. it is true that the modalities like acoustic and visual often provide more evidences about the context of the utterance in comparison to text. for sarcasm detection, the very first multimodal dataset named as mustard has been very recently released by @xcite , where the authors used a support vector machine (svm) classifier for sarcasm detection. in our current work, we at first extend the mus-tard dataset @xcite by manually labeling each utterance with sentiment and emotion labels. thereafter, we propose a deep learning based approach along with two attention mechanisms (i e -attention and i a -attention) to leverage the sentiment and emotion for predicting sarcasm in a multi-modal multi-task framework. further, to the best of our knowledge, this is the very first attempt at solving the multi-modal sarcasm detection problem in a deep multi-task framework. we demonstrate through a detailed empirical evaluation that sarcasm detection can be improved significantly if we are successful in leveraging the knowledge of emotion and sentiment using an effective multi-task framework. ## dataset the mustard @xcite dataset consists of conversational audio-visual utterances (total of 3.68 hours in length). this dataset consists of 690 samples, and each sample consists of utterance accompanied by its context and sarcasm label. the samples were collected from 4 popular tv series viz., friends, the big bang theory, the golden girls, and sarcasmaholics anonymous and manually annotated for the sarcasm label. the dataset is balanced with an equal number of samples for both sarcastic and non-sarcastic labels. the utterance in each sample consists of a single sentence, while the context associated with it consists of multiple sentences that precede the corresponding utterance in the dialogue. we manually re-annotated this dataset to introduce sentiment and emotion labels in addition to sarcasm. we define two kinds of emotion and sentiment values viz., implicit and explicit, which are discussed in the following subsections. ## proposed methodology in this section, we describe our proposed methodology, where we aim to leverage the multi-modal sentiment and emotion information for solving the problem of multi-modal sarcasm detection in a multi-task framework. we propose a segment-wise inter-modal attention based framework for our task. we depict the overall architecture in figure 4 . the extended dataset with annotation guidelines and source code are available at http://www.iitp.ac . each sample in the dataset consists of an utterance (u) accompanied by its context (c) and labels (sarcasm, implicit sentiment, explicit sentiment, implicit emotion, and explicit emotion). the context associated with the utterance consists of multiple sentences (say, n) that precede the corresponding utterance in the dialogue. each utterance and its' context is associated with its' speaker i.e., speaker of utterance (sp u ) and speaker of context (sp c ), respectively. we represent sp u and sp c by using a one-hot vector embedding. we divide our proposed methodology into three subsections i.e., input layer, attention mechanism and output layer, which are described below: ## experiments and analysis we divide the whole process into four categories: i). utterance without context without speaker (i.e., we do not use the information of context and its' speaker with utterance); ii). utterance with context without speaker (i.e., we use the context information with utterance but not speaker information); iii). utterance without context with speaker (i.e., we use the speaker information with utterance but not context information); and iv). utterance with context with speaker (i.e., we use the context and its' speaker information with utterance). ## conclusion in this paper, we have proposed an effective deep learning-based multi-task model to simultaneously solve all the three problems, viz. sentiment analysis, emotion analysis and sarcasm detection. as there was no suitable labeled data available for this problem, we have created the dataset by manually annotating an existing dataset of sarcasm with sentiment and emotion labels. we have introduced two attention mechanisms (i.e., i e -attention and i a -attention), and incorporated the significance of context and speaker information w.r.t. sarcasm. empirical evaluation results on the extended version of the mustard dataset suggests the efficacy of the proposed model for sarcasm analysis over the existing state-of-the-art systems. the evaluation also showed that the proposed multi-tasking framework achieves better performance for the primary task, i.e. sarcasm detection, with the help of emotion analysis and sentiment analysis, the two secondary tasks in our setting. during our analysis, we found that the dataset is not big enough for a complex framework to learn from. along with investigating new techniques, we hope that assembling a bigger curated dataset with quality annotations will help in better performance.