A Sentiment and Emotion Aware Multimodal Multiparty Humor Recognition in Multilingual Conversational Setting (2022)

Abstract: In this paper, we hypothesize that humor is closely related to sentiment and emotions. Also, due to the tremendous growth in multilingual content, there is a great demand for building models and systems that support multilingual information access. To end this, we first extend the recently released Multimodal Multiparty Hindi Humor (M2H2) dataset by adding parallel English utterances corresponding to Hindi utterances and then annotating each utterance with sentiment and emotion classes. We name it Sentiment, Humor, and Emotion aware Multilingual Multimodal Multiparty Dataset (SHEMuD). Therefore, we propose a multitask framework wherein the primary task is humor detection, and the auxiliary tasks are sentiment and emotion identification. We design a multitasking framework wherein we first propose a Context Transformer to capture the deep contextual relationships with the input utterances. We then propose a Sentiment and Emotion aware Embedding (SE-Embedding) to get the overall representation of a particular emotion and sentiment w.r.t. the specific humor situation. Experimental results on the SHEMuD show the efficacy of our approach and shows that multitask learning offers an improvement over the single-task framework for both monolingual (4.86 points in Hindi and 5.9 points in English in F1-score) and multilingual (5.17 points in F1-score) setting.

## introduction humor @xcite @xcite is an essential aspect of daily conversation, and people try to provoke humor in their talks. @xcite defined humor as "the nature of experiences to induce laughter and provide amusement". humor is a tool by which anyone can draw the audience's attention. even in a formal conversation, humor may make a person look more attractive and thus may lead to a better conclusion. irrespective of its relation to intelligence, humor is inherently a challenging problem to understand. to understand humor, we also take some additional information into consideration i.e., sentiment @xcite @xcite and emotion @xcite @xcite to help humor detection in conversations. sentiment and emotion are two aspects that help each other, which has already been shown in @xcite . we hypothesize that sentiment and emotion do not only help each other @xcite but also help humor. for example, it is difficult to decide whether the following utterance "a º ú я । (oh in old age i would have taken you myself)" is humorous or not. but, careful observation of the sentiment (positive) and emotion (happy) of the speaker helps us understand that the speaker is in a funny mood and trying to mock his wife. sometimes it is difficult for the global audience to understand any local language like hindi, so this is where multilingual comes into the picture. also, multilinguality provides freedom for the model to understand humor; e.g., if the model is unable to understand the one language's word (say hindi) for a particular utterance, then other languages (say english) can give a better clue to understand humor. thus, we propose a multitask framework in a multilingual setting wherein the primary task is humor detection, and the auxiliary tasks are sentiment and emotion identification. we have already discussed that sometimes just going through the utterance text is not enough to understand humor. the other modalities, such as visual and acoustic, can provide additional cues. in natural language processing, understanding humor from these modalities comes under the boundaries of multimodal language. humor may be found in almost every human-to-human encounter. some works @xcite @xcite @xcite @xcite @xcite @xcite on multimodal humor have already been done. @xcite proposed the ur-funny dataset to aid in the comprehension of multimodal language utilized in the expression of humor. the author has also shown the importance of multimodality over unimodality. humor can be created with any modality. @xcite looked at how humor was created using only verbal media, both verbal and visual media, and exclusively visual media. the data was gathered from 74 political comic strips published in kompas newspaper. the author used the general theory of verbal humor (gtvh) for humor analysis. higher pitch or loudness, a faster speaking cadence, or considerable pauses are not characteristics of conversational humor. in the paper, @xcite discovered that when people are laughing or smiling, they are more likely to be humorous. el @xcite did two experiments: the first experiment looks at whether untrained people can recognize the structural aspects of humor in a multimodal text. the second research investigates the sequence in which textual and visual inputs are processed (tweets). in another work, @xcite discussed the quality of a conversation and how conversation partners see one other and are influenced by irony and irony-related humor. the humor styles questionnaire (hsq) distinguishes four types of humor that can be good or damaging to one's mental health. @xcite studied to compile research that used the hsq to analyze the relationships between distinct humor types and four distinct aspects of mental health (self-esteem, life satisfaction, optimism, depression). @xcite proposed multimodal multiparty hindi humor (m2h2) dataset for conversations which was the very first of its kind. then, the authors employed two strong baseline setups, viz. misa foot_0 w/ dialoguernn and misa w/ bclstm. they used misa for fusion and dialoguernn & bclstm for understanding the contextual relationship among words. in comparison to the existing systems, we first created the shemud by manually annotating each utterance in the english language and then labeling each utterance with sentiment and emotion classes. after that, we design a multitasking framework wherein we first propose a context transformer to capture the deep contextual relationships with the input utterances. we then propose a sentiment and emotion aware embedding (se-embedding) to get the overall representation of a particular emotion and sentiment w.r.t. the specific humor situation. further, to the best of our knowledge, this is the first attempt at solving the sentiment and emotion aware humor detection in a multilingual conversational setting. empirical results on the shemud dataset demonstrate that the baselines yield better performance over the state-of-the-art systems. we extend the m2h2 dataset (chauhan et al., 2021 ) by manually annotating each hindi utterance with the english language, making it a multilingual dataset (shemud), and then annotating each utterance with sentiment and emotion classes. for sentiment annotation, we consider three sentiment classes, namely positive, negative and neutral. while for emotion annotation, we annotate the dataset with seven emotion values 2 , viz. anger, disgust, fear, happy, sad, surprised, and neutral (c.f. ## proposed methodology this section describes our proposed model, where we aim to leverage multimodal sentiment and emotion information for solving multimodal humor detection in a multitask framework. we depict the overall architecture in figure 1 . the extended dataset and source code are available at https://www.iitp.ac.in/ ~ai-nlp-ml/resources.html#shemud ## experimental setup we evaluate our proposed model on the shemud. we perform a grid search to find the optimal hyperparameters, which are shown in ## comparative analysis we compare the results obtained from our proposed model against the existing model m2h2 @xcite . we evaluate our proposed framework with all the possible combinations i.e., unitask (h), please note that as the presented model in m2h2 (2021) was the only unitask (h) model without multilingual, so we made some changes in the model to make it a multitask model and multilingual. thereafter we obtain the results for unitask (h), bitask (h+s and h+e) and tritask (h+s+e) for both monolingual and multilingual which are also shown in ## ablation study we perform an ablation study to show the efficacy of se-embedding for both monolingual and multilingual. for monolingual, we perform experiments with and without se-embedding. we show the ablation experimental results in similarly, for multilingual, we perform experiments with and without se-embedding. as per the result (c.f. ## error analysis we perform a detailed analysis to realize the model's strengths and weaknesses. to better analyze and justify our proposed model in terms of quality, we use the same samples (for multilingual setup), which were wrongly predicted and depicted in the paper m2h2 (2021). we show these samples in ## conclusion in this paper, we have proposed an effective deep learning-based multitask model for humor detection (primary task) with auxiliary tasks, i.e., sentiment and emotion, in a multilingual conversational setting. as there was no suitable data available for this problem, we first extend recently released m2h2 dataset by adding parallel english utterances corresponding to hindi utterances and made it a multilingual dataset. then, we annotate each utterance with sentiment and emotion classes. we have proposed a multitasking framework wherein we propose a context transformer to capture the deep contextual relationships with the input utterances. we have also successfully established a sentiment and emotion embedding (se-embedding) which gets the overall representation of a particular emotion and sentiment w.r.t. the specific humor situation. experimental results on the shemud have shown that the multitask system yields better performance over the single-task framework. in the future, we would like to extend our work towards the multiparty dialogue generation in hindi with the help of humor, sentiment, emotion and speaker information.