ISDS - NLP at S em E val-2024 Task 10: Transformer based neural networks for emotion recognition in conversations (2024)

Abstract: This paper outlines the approach of the ISDS-NLP team in the SemEval 2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF). For Subtask 1 we obtained a weighted F1 score of 0.43 and placed 12 in the leaderboard. We investigate two distinct approaches: Masked Language Modeling (MLM) and Causal Language Modeling (CLM). For MLM, we employ pre-trained BERT-like models in a multilingual setting, fine-tuning them with a classifier to predict emotions. Experiments with varying input lengths, classifier architectures, and fine-tuning strategies demonstrate the effectiveness of this approach. Additionally, we utilize Mistral 7B Instruct V0.2, a state-of-the-art model, applying zero-shot and few-shot prompting techniques. Our findings indicate that while Mistral shows promise, MLMs currently outperform them in sentence-level emotion classification.

## introduction task 10 from semeval 2024 competition @xcite addresses the complex challenge of identifying the emotions within dialogues (english and hindi). this task comprises two primary objectives: firstly, assigning an emotion label to each utterance within a dialogue, and secondly, discerning the trigger utterance or utterances responsible for an emotion-flip within the dialogue @xcite . emotions play a crucial role in human interaction and one can understand more from a text if one knows the underlying sentiment of the writer. in contexts where disagreements may arise, such as customer service platforms, virtual assistant chats or forums, identifying trigger utterances for emotion flips can help mediate conflicts and prevent escalation. a chatbot dealing with an angry customer would benefit from knowing how to speak in order to generate empathetic responses. if it knows that the chatbot's current sentence can trigger an emotion flip from neutral to anger, the chatbot should refine it, or if the emotion flip is from anger to joy, the chatbot should be more confident in such a response in the future. both types of models we tried for subtask 1 were based on transformers. the first one used bertlike models and we achieved the best accuracy with them, while the second one is a state of the art causal model (mistral, @xcite ) that was tested in zero-shot and few-shot settings with poorer results. although in the first task our system worked well, placing 12th in the leaderboard, the other 2 tasks were much harder and we placed 14th on the second subtask. we believed that with a better strategy to prevent overfitting (like under or oversampling), our system would have improved. our code is open source and available to use on github. ## background the competition had 3 subtasks explained in figure 1 and we participated in all of them with the best results on subtask 1 where we placed 12th with an f1 score of 0.43. ## system overview we tried two approaches, both of them based on transformer architecture: masked language modelling and casual modelling. we chose these two architectures because of their recent successes in nlp. ## results our top-performing model ( this highlights the superiority of fine-tuned masked language models (mlms) over mistral for sentence classification tasks. the results suggest that smaller causal models remain less effective than fine-tuned mlms in this domain. we also see that few-shot mistral is worse than zeroshot, probably because too much data in the prompt confuses the model. model train validation test xlm-roberta 0.74 0.57 0.43 mdeberta-v3 0.74 0.56 0.42 bert-multi 0.66 0.48 0.35 mistral zero-shot --0.32 mistral few-shot --0.31 distilbert-multi 0.6 0.47 0.29 in terms of number of epochs, our best model was overfitting when finetuned for too many epochs ( ## conclusion overall, our system achieved encouraging results in subtask 1, despite exhibiting some overfitting for dominant labels. while performance on the emotion-flip detection tasks (subtasks 2 and 3) highlights areas for improvement, we still placed in the first half of the leaderboard. looking ahead, we plan to investigate hybrid transformer-lstm architectures for a more nuanced understanding of emotion-flip triggers. additionally, enriching the data by incorporating a broader conversational context through multi-turn analysis could enhance our model's capabilities. not least, even though we tried mistral, there are newer causal models like mixtral @xcite and solar @xcite which could perform better at this type of task.