T rans M istral at S em E val-2024 Task 10: Using Mistral 7 B for Emotion Discovery and Reasoning its Flip in Conversation (2024)

Abstract: The EDiReF shared task at SemEval 2024 comprises three subtasks: Emotion Recognition in Conversation (ERC) in Hindi-English code-mixed conversations, Emotion Flip Reasoning (EFR) in Hindi-English code-mixed conversations, and EFR in English conversations. The objectives for the ERC and EFR tasks are defined as follows: 1) Emotion Recognition in Conversation (ERC): In this task, participants are tasked with assigning an emotion to each utterance within a dialogue from a predefined set of possible emotions. The goal is to accurately recognize and label the emotions expressed in the conversation; 2) Emotion Flip Reasoning (EFR): This task involves identifying the trigger utterance(s) for an emotion-flip within a multi-party conversation dialogue. Participants are required to pinpoint the specific utterance(s) that serve as catalysts for a change in emotion during the conversation. In this paper we only address the first subtask (ERC) making use of an online translation strategy followed by the application of a Mistral 7B model together with a few-shot prompt strategy. Our approach obtains an F1 of 0.36, eventually exhibiting further room for improvements.

## introduction affective computing has experienced a resurgence, largely propelled by recent advancements in artificial intelligence. emotion recognition in conversations (erc) has emerged as a prominent task within affective computing, garnering increasing attention @xcite . its objective is to discern the emotion conveyed in each utterance during conversations, with implications for various applications including the development of effective dialogue systems, facilitating social viewpoint mining, and creating intelligent medical systems. current research in erc primarily focuses on capturing the emotional state of speakers through contextual analysis and establishing distinct contexts for different speakers, often leveraging multimodal data to support this endeavour. despite recent strides, two major challenges persist: the increasing demand for automated tools capable of extracting and categorizing data from online sources underscores the need to address both established and emerging societal concerns efficiently. recent strides in machine and deep learning architectures have sparked significant interest in natural language processing (nlp). efforts have been intensified towards developing techniques for automating the identification and categorization of textual content prevalent on the internet today. in the literature, various strategies have been proposed for performing text classification tasks. over the past fifteen years, some of the most successful strategies have included support vector machines (svm) @xcite , convolutional neural networks (cnn) @xcite , graph neural networks (gnn) @xcite , ensemble models @xcite , and more recently, transformers @xcite . at semeval-2024 task 10 (kumar et al., 2024a) -emotion discovery and reasoning its flip in conversation (ediref) -three subtasks were proposed. all the three subtasks are presented and discussed in the section 2. to face with the first subtask (erc), we proposed a transformer-based approach which made use of mistral 7b @xcite . we used the model in a particular few-shot way described in the rest of this paper. specifically, after translating all the code-mixed samples in english, we provided the samples from the labelled train and dev set to the model, asking while prompting to predict the emotion for the current utterance (i.e., the current sample from the test set). the rest of the paper is made as follows. in section 2 we provide some background on the task 10 hosted at semeval-2024. in section 3 we provide a description of the approach presented. in section 4 we provide details about the experimental setup to replicate our work. in section 5, the results of the official task and some discussions are provided. in section 6 we present our conclusion and proposals for future works. we make all the code publicly available and reusable on github foot_0 . ## background this section furnishes background information regarding task 10 (subtask 1), held at semeval-2024. the ediref shared task at semeval 2024 (kumar et al., 2024a) encompasses three distinct subtasks, namely: emotion recognition in conversation (erc) within hindi-english code-mixed conversations, emotion flip reasoning (efr) within hindi-english code-mixed conversations, and efr within english conversations @xcite . the erc task involves the assignment of emotions to each utterance within a dialogue, drawn from a predefined set of potential emotions. conversely, the efr task focuses on identifying the trigger utterance(s) responsible for inducing an emotion-flip within a multi-party conversation dialogue. in the figure 1 , is reported a sample from the official competition website foot_1 and specifically related to the subtask 1 (i.e., erc). for subtask 1, a submission entails a singular json file with each emotion in a new line. every emotion correspond to each utterance in the official provided test set. the second and the third subtasks differ on the in this case, we report samples related to the first erc task in which we took part. figure 2 : example of some samples from the dataset. in this case, we report samples related to the efr task. ## system overview while it has been established that transformers may not always be the optimal choice for text classification tasks @xcite , the efficacy of various strategies, such as domain-specific finetuning @xcite and data augmentation @xcite , depends on the specific objectives. the increasing adoption of transformer-based architectures in academic research has also been bolstered by various methodologies showcased at semeval 2024. these methodologies tackle di- train+dev sets test set deep translator deep translator mistral 7b cms few-shot "hailo varld" en few-shot "hello world" joy figure 3 : the system overview of our proposed approach. given a set of code mixed samples (i.e., cms) from the train and dev sets, they are translated to english using deep translator (i.e., english samples). then they are all provided as input -a few-shot samples from the training set, together with the emotion definitions -to mistral 7b. following these few shot samples and the definitions as input, there is one utterance from the test set for which the emotion has to be predicted. our approach is few-shot (littenberg-tobias et al., 2022) and make use of mistral 7b. mistral 7b, a language model boasting 7 billion parameters, is engineered to excel in both performance and efficiency. in comparison to the leading open 13b model (llama 2), mistral 7b demonstrates superior performance across all assessed benchmarks. furthermore, it surpasses the top released 34b model (llama 1) in tasks related to reasoning, mathematics, and code generation. the model capitalizes on grouped-query attention (gqa) to expedite inference, complemented by sliding window attention (swa) to effectively process sequences of varying lengths while minimizing inference costs. additionally, a fine-tuned variant, mistral 7b -instruct, is tailored for adhering to instructions, and it outperforms llama 2 13b -chat model across both human and automated benchmarks. the introduction of mistral 7b instruct underscores the ease with which the base model can be fine-tuned to achieve notable performance enhancements. for our task, before prompting the model with the current sample from the test set, we made an online and real-time use of google translator from the deep_translator 3 library. then we randomly selected eighty samples from the provided labelled training set and other eighty from the provided labelled dev set. then we formatted the samples in each set in the following way: ## results as described on the official task page foot_5 , the evaluation criteria for the three tasks are delineated as follows: to generate the prediction file: t1-f1 t2-f1 t3-f1 transmistral 7b 0.36 0.10 0. 22 then, it was asked to aggregate the outputs from all tasks into a single file named 'answer.txt', structured as follows: upon compilation, the organizers asked to create a zip file encompassing 'answer.txt' and proceed with submission as per guidelines. in the the ## conclusion this paper presents the application of mistral 7bmodel for addressing the task 10 at semeval-2024. for our submission, we decided to follow few-shot learning approach, employing as-is, an in-domain pre-trained transformer. after several experiments, we found it beneficial to build a prompt containing some samples from the training and from the dev set. then we provide as a prompt the current translated sample together with the few-shot samples. the model was asked to select one of the emotion among the ones available. the task presents inherent challenges, with evident scope for refinement, as underscored by the final ranking. potential alternative methodologies encompass leveraging the zero-shot capabilities inherent in other models such as gpt and t5, expanding the training set size through the incorporation of additional data, or adopting alternative strategies for integrating ontology-based domain knowledge beyond the approaches delineated in our study. furthermore, refinement opportunities exist through fine-tuning and recontextualizing the problem as a text classification task. furthermore, given the interesting results recently provided on a plethora of tasks, also other few-shot learning @xcite @xcite or data augmentation strategies @xcite @xcite could be employed to improve the results. looking at the final ranking, our simple approach exhibits some room for improvements. however, it is worth notice that it required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by google colab. also, thanks to the proposed approach, we have been able to outperform the baseline provided by the task organizers.