MEISD : A Multimodal Multi-Label Emotion, Intensity and Sentiment Dialogue Dataset for Emotion Recognition and Sentiment Analysis in Conversations (2020)

Abstract: Emotion and sentiment classification in dialogues is a challenging task that has gained popularity in recent times. Humans tend to have multiple emotions with varying intensities while expressing their thoughts and feelings. Emotions in an utterance of dialogue can either be independent or dependent on the previous utterances, thus making the task complex and interesting. Multi-label emotion detection in conversations is a significant task that provides the ability to the system to understand the various emotions of the users interacting. Sentiment analysis in dialogue/conversation, on the other hand, helps in understanding the perspective of the user with respect to the ongoing conversation. Along with text, additional information in the form of audio and video assist in identifying the correct emotions with the appropriate intensity and sentiments in an utterance of a dialogue. Lately, quite a few datasets have been made available for dialogue emotion and sentiment classification, but these datasets are imbalanced in representing different emotions and consist of an only single emotion. Hence, we present at first a large-scale balanced Multimodal Multi-label Emotion, Intensity, and Sentiment Dialogue dataset (MEISD), collected from different TV series that has textual, audio and visual features, and then establish a baseline setup for further research.

## introduction with the advancements in artificial intelligence (ai), the gap between natural language processing (nlp) and computer vision (cv) has been bridged by extensive research in multi-modal information analysis. the ability to use different modalities such as text, audio and video for different tasks, such as emotion classification @xcite , sentiment analysis @xcite , dialogue generation @xcite have helped in building robust systems. the potential to understand correct emotion and sentiment in a conversation is crucial for developing strong human-machine interaction systems. dialogue systems are of two types i.e., goal-oriented systems @xcite or open chit-chat systems @xcite . in both these systems, understanding the user's emotions is crucial to maximizing the user experience and satisfaction. nowadays, there is a huge demand for developing social agents capable of having real conversations with humans. with the rapid growth in technology, personal assistants in smartphones such as amazon's alexa, apple's siri, and google's home have become human companions. hence, these applications need to understand the correct emotional state of the user to increase user contentment leading to user retention. emotions and sentiments are subjective qualities and are understood to share overlapping features; hence are frequently used interchangeably. this is mainly because both sentiment and emotion refer to experiences resulting from the combination of biological, cognitive, and social influences. though both are considered to be the same, yet according to @xcite , the sentiment is formed and retained for a longer duration, whereas emotions are like episodes that are shorter in length. moreover, the sentiment is mostly target-centric, while emotions are not always directed to a target. previously, sentiment and emotions have been tackled separately, although they are different but closely related. lately, emotion detection and sentiment analysis in multimodal systems using audio, video, and textual features have gained popularity. but both these tasks have not been explored in depth for conversations. the main reason for this is the unavailability of a large-scale multi-modal dialogue dataset labeled with emotions and sentiment to facilitate research in this direction. also, identifying emotions and sentiments in conversations is a challenging task compared to tweets or sentences. this is mainly because the contextual information or past utterances may influence the emotions of the present utterance. also, emotional state change among the speakers in a conversation makes it difficult to identify the emotions and sentiment of an utterance in a dialogue. with the release of multimodal emotionlines dataset (meld), research in emotion and sentiment identification in conversations has gained immense attention. this dataset comprises the conversations taken from the friends tv series labeled with sentiment and emotion using text, audio, and video information. the dataset provides multimodal information for classifying emotions and sentiments in dialogues. this dataset is made using a comedy tv series; it is unbalanced in its emotion distribution, making the dataset imbalanced. human emotions are extremely complex; therefore, it is highly probable that they express multiple emotions in a single utterance. there is a huge possibility that multiple emotions expressed in an utterance are correlated. for example, the speaker may express the emotion "anger" and "disgust" often together than in isolation. also, the intensity of the different emotions in a given utterance may vary. for example, the speaker, in some cases, express "anger" with higher intensity while "disgust" with lower intensity or vice-versa. the meld dataset is labeled with a single emotion only, thereby not providing the complete emotional information in a given utterance. for building robust emotion and sentiment classification systems, it is crucial to have a balanced dataset labeled with sentiment and multiple emotions along with their corresponding intensity to provide the complete affective information of a given utterance. hence, in this work, we propose a large-scale balanced multimodal multi-label emotion, intensity, and sentiment dialogue (meisd) dataset labeled with multiple emotions, intensity, and sentiments using textual, audio, and visual information, collected from 10 tv series belonging to different genres. only textual information is not enough for understanding emotions, as emotion is also expressed through facial expressions, gestures, pitch, and tone. for example, the given utterance "great, you are here" can exhibit different emotions, such as joy, anger, or surprise. hence, it is difficult to identify the correct emotion using only the textual information. hence, the sentiment label of these utterances is also ambiguous. it is essential to simultaneously focus on these utterances' audio and visual counterparts for identifying the correct emotions and sentiment label of these utterances. an example of a conversation from the meisd dataset labeled with sentiment and multiple emotions, and their corresponding intensity is given in figure 1 . as it is evident from the given example, visual information provides additional knowledge for determining the correct emotions and sentiment labels. to the best of our knowledge, this is the first dialogue data labeled with multiple emotions, intensity, and sentiment for identifying emotions and sentiments in conversations and will hopefully promote further research in this area. the major contributions of our present work are: the rest of the paper is structured as follows. in section 2, we present a brief survey of the related work. in section 3, we describe the details of the dataset that we create. in section 4, we explain the methodology. the experimental setup, along with the evaluation metrics, is reported in section 5. in section 6, we present the results along with the necessary analysis. finally, we conclude in section 7 with future work directions. ## related work most of the early research on emotion classification and sentiment analysis was performed separately upon textual datasets mostly taken from twitter @xcite @xcite @xcite . in @xcite , the authors proposed a rnn framework capable of learning inter-modal interaction among the different modalities using the auto-encoder mechanism. as emotion and sentiment are two very closely related tasks, in recent time there is a trend on modeling both sentiment and emotion of an utterance simultaneously @xcite @xcite . in @xcite , the authors employed the concept of multi-task learning for multi-modal affect analysis and explored a contextual inter-modal attention framework that aimed in leveraging the association among the neighboring utterances and their multi-modal information. with the advancements in artificial intelligence (ai), emotion classification and sentiment analysis have become a significant task due to its importance in many downstream tasks, such as customer behavior modeling, response generation for conversational agents, multimodal interactions etc. hence, to maximize user satisfaction and providing a better experience to the customer, it is important to understand the correct emotion and sentiment of the customer. recently, multi-label emotion classification has been investigated for textual data in @xcite @xcite . using multiple convolution neural network (cnn) networks along with self-attention, the authors in @xcite performed multi-label emotion classification on twitter data. similarly, the authors in @xcite improved the performance of multi-label emotion classification on twitter data by using transfer learning. lately, sequence-to-sequence framework @xcite has been employed for multi-label emotion classification. our present work differs from these single and multi-label emotion and sentiment classification works as we tend to classifying emotions and sentiments on dialogue conversations that require contextual information of the previous utterances, thereby making the task more challenging and interesting. every human-machine interactions are grounded in conversations driven by emotions. hence, identifying the emotion in dialogue is essential for building robust systems capable of such interactions. recently, investigations on emotion detection in conversations has been in demand. the authors in @xcite released a dataset taken from friends tv series for detecting emotions in dialogues. similarly, in @xcite an attention framework was designed for identifying emotions in spoken dialog systems. in @xcite , memory networks were adopted to capture contextual information for emotion detection in conversations. to capture the contextual information in conversations, dia-loguernn @xcite employs three gated recurrent units (gru) for effectively modeling the past utterances of the speaker and the listener in dyadic conversations for emotion detection. as conversation itself is multimodal, people involved in conversations use various facial expressions, gestures and different pitch, tones to emote their feelings making the conversation dependent on the audio and visual aspect as well. hence, quite a few multimodal datasets have been employed to identify emotion using audio and visual information as well. in @xcite , the author proposed an interactive memory network that extracts multimodal features for emotion classification. iemocap dataset @xcite has been used for emotion detection using a deep neural framework that uses the multimodal information at the final layer for emotion identification. multimodal sentiment analysis has also been investigated for correct classification of sentiments @xcite . the authors in @xcite proposed a novel hierarchical feature fusion strategy for integrating different modalities, such as audio, video and text for identifying the sentiments. the authors in @xcite extended the emotionlines dataset by incorporating audio and visual modalities for correct identification of emotions and sentiments in conversations. the meld dataset has been further used for building different neural frameworks for jointly identifying emotion and sentiment from conversations @xcite @xcite . as opposed to these existing works on multimodal emotion and sentiment classification on dialogue data, our present works provides a balanced multimodal multi-label emotion, intensity and sentiment dataset for the classification of multiple emotions and sentiment in the given utterance. ## multimodal multi-label emotion, intensity and sentiment dialogue (meisd) dataset we create the meisd dataset foot_0 from the 10 famous tv shows belonging to different genres: (i). comedy: friends, the big bang theory, how i met your mother, the office; (ii). drama: house m.d., grey's anatomy, castle and game of thrones, house of cards, breaking bad. this dataset consists of conversations with utterances from multiple speakers making it a multi-party conversational dataset. the dataset contains dialogues mostly from all the episodes belonging to the different seasons of the tv series giving us a wide variation in dialogues. in total, we have 1000 dialogues from all the tv series in our dataset. firstly, we obtain the start and end timestamps of every dialogue from the different episodes of the tv series. we extract all the subtitles and transcripts for every dialogue with their respective timestamps. thereafter, we segment the dialogues into utterances following the heuristics similar to @xcite : (i). the timestamps of the utterances belonging to a dialogue should always be in the increasing order; (ii). the utterances in a particular dialogue should be from the same episode only. utterances in the subtitles were sometimes grouped together under the same timestamp in the subtitle files. hence, we use the transcription alignment tool gentle foot_1 for extracting the accurate timestamp information of every utterance as it automatically aligns the text with the audio by obtaining the word-level timestamp information from the audio file. after extracting the corresponding timestamps of every utterance in a dialogue, we then obtain their audio and visual clips from the source episodes. after getting the audio and visual clips of every utterance, we extract the audio and visual files from these clips. the audio files are then formatted as 16-bit pcm wav files for further processing. the video files were used to extract 2048d pooled features using the last convolution block of resnet101. our final meisd dataset comprises of textual, visual and audio features that bring the three important modalities together for effective multi-label emotion, intensity and sentiment analysis. ## experiments the extraction of features along with the details of the baseline models to evaluate our proposed meisd dataset is described in this section. we also discuss the metrics used to evaluate the models on the proposed dataset. ## result and discussion in this section, we provide the results for all the three tasks, i.e. multi-label emotion classification, intensity prediction and sentiment analysis on our proposed meisd dataset. in from the table, we can also infer that using solely the audio and video features of every utterance decreases the performance of the model in identifying the correct emotions. the major information about the emotions is achieved from the textual features itself, hence the performance of the models using only textual features is far better than the models having only audio and video features as input. while using all the features, they together boost the performance of the model. hence, it can be concluded that the audio and visual counterparts of an utterance assist in identifying the correct emotions of a particular utterance. since in our final baseline model we only enhance the performance by using better textual representation, hence the performance on audio and visual are similar to the dialoguernn baseline. for the intensity prediction task, we report the pearson correlation co-efficient as a metric and from the table it is visible that the final proposed baseline yields the highest score of 0.588 using information from all the three modalities. simultaneously, in ## conclusion and future work in this paper, we have introduced a large-scale multimodal multiparty conversational dataset, meisd for multi-label emotion classification, intensity prediction and sentiment analysis in conversations. the detailed description of the dataset along with the entire process for building the dataset has been discussed in the paper. meisd dataset is a multimodal dataset that has textual, audio and visual features for every utterance of dialogue taken from 10 different tv series belonging to the different genres, thereby providing a large diversity to the meisd dataset. hence, this dataset provides diversity with respect to utterances, scene information, characters and emotional expressions, and hence offer a wide variety in dialogues and make the task all the more challenging. we have evaluated our proposed meisd datasets and reported the results using strong baselines for all the three tasks of emotion recognition, intensity prediction and sentiment classification. we believe that this dataset can be employed in the future for multi-label emotion, intensity and sentiment detection in conversations. in the future, this dataset can be employed for multi-task learning of all the three tasks simultaneously in dialogues. as all the three tasks are closely related, hence through multi-task learning the performance of the tasks might improve due to the shared information. this dataset can also be used for building emotional and sentimental conversational agents. furthermore, the multimodality aspect of the dataset can be investigated deeply employing different fusion techniques for achieving better multi-modal interactions that can help in the tasks. also, research on novel frameworks for capturing the contextual information