L ast R esort at S em E val-2024 Task 3: Exploring Multimodal Emotion Cause Pair Extraction as Sequence Labelling Task (2024)

Abstract: Conversation is the most natural form of human communication, where each utterance can range over a variety of possible emotions. While significant work has been done towards the detection of emotions in text, relatively little work has been done towards finding the cause of the said emotions, especially in multimodal settings. SemEval 2024 introduces the task of Multimodal Emotion Cause Analysis in Conversations, which aims to extract emotions reflected in individual utterances in a conversation involving multiple modalities (textual, audio, and visual modalities) along with the corresponding utterances that were the cause for the emotion. In this paper, we propose models that tackle this task as an utterance labeling and a sequence labeling problem and perform a comparative study of these models, involving baselines using different encoders, using BiLSTM for adding contextual information of the conversation, and finally adding a CRF layer to try to model the inter-dependencies between adjacent utterances more effectively. In the official leaderboard for the task, our architecture was ranked 8th, achieving an F1-score of 0.1759 on the leaderboard.

## introduction emotion analysis is one of the fundamental and earliest sub-fields of nlp that focus on identifying and categorizing emotions that are expressed in text. earlier, research in this domain focused on emotion detection in news articles and headlines @xcite . however, later emotion recognition in conversation gained popularity due to the widespread availability of public conversation data @xcite . recently, the task of emotion cause analysis has gained traction, which tries to identify the causes behind certain emotions @xcite . this has widespread application such as building chatbots that can identify the emotions of the user and even identify the cause behind the emotions to perform certain actions @xcite . for instance, companies can identify causes behind dissatisfaction in customer interactions and take appropriate measures @xcite , ai-driven therapeutic insights can be gained using such models (dalfonso, 2020), social media content moderation can be better done @xcite , work management and team management by companies can be improved @xcite . in the task @xcite , we tackle the problem of multimodal emotion cause pair extraction, where given a set of utterances in a conversation, we must identify the following: our proposed system tackles the task in a 3-step fashion -(a) first, we train a model to identify the emotions that are expressed in individual utterances in a conversation. (b) next, we train a model to identify whether an utterance can be a cause of an emotion expressed in another/same utterance (candidate causes). (c) finally, we train a model to pair emotion-utterances with their causes among the possible candidate causes. for both the (a) and (b) models we experiment with 3 basic architectures -(i) a simple neural network to determine the class of emotion (n-class classifier) and another neural network to identify whether the utterance is a candidate cause or not (binary classification). (ii) a bilstm @xcite architecture that accounts for the surrounding context 1 1204 of the conversation while doing the n-class and binary-classification. (iii) a bilstm crf @xcite architecture which accounts for the surrounding emotions as well while doing the n-class classification. we also experiment with different encoders for the three modalities. ## experimental setup we perform a random shuffle and use a 90-10% split for the train-validation split. the test set was provided by the authors, but its gold labels have not been made public. the experiments involving baseline ii and iii use emotionroberta + wavlm + mvitv2 configuration. all the experiments involve applying a dropout of 0.3 on the audio, visual and textual embeddings before they are passed on to the main architectures. the bilstm for emotion detection consists of 4 layers while the one for candidate cause identification contains 3 layers. the dropout between the stacked layers of the bilstm is kept 0.3 as well. we use adamw optimizer for all the three models, and use a linear learning rate scheduler with warmup for training the models. the emotion classification model is trained for 60 epochs, the candidate cause identification model is trained for 40 epochs, and the emotion-cause pairing model is trained for 40 epochs as well. in order to train the emotion-cause pairing model, we create positive and negative pairs during training. however, while the number of positive pairs is of the order n, the number of negative 4 1207 pairs comes to the order of n 2 , and thus we perform a random sampling of the negative pairs to keep the positive and negative samples in the ratio 1:5. this helps us to maintain balance between the positive and negative classes. ## results and analysis the performance of the three baselines can be seen in ## conclusion in conclusion, we observe that the utterance labeling systems perform as good as sequence labeling systems for this specific dataset. further, we also see that encoders which are trained on other emotion-related tasks tend to perform better on similar emotion-related tasks. in future, it is possible to learn joint embeddings over the 3 modalities, which should provide better representations for each utterance @xcite . further, it can be experimented to utilize the speaker information for each utterance while creating utterance representations @xcite .