RACAI at S em E val-2024 Task 10: Combining algorithms for code-mixed Emotion Recognition in Conversation (2024)

Abstract: Code-mixed emotion recognition constitutes a challenge for NLP research due to the text’s deviation from the traditional grammatical structure of the original languages. This paper describes the system submitted by the RACAI Team for the SemEval 2024 Task 10 - EDiReF subtasks 1: Emotion Recognition in Conversation (ERC) in Hindi-English code-mixed conversations. We propose a system that combines a transformer-based model with two simple neural networks.

## introduction emotion recognition in conversation (erc) @xcite is a crucial task in conversational artificial intelligence research that aims to identify the emotion of each utterance in a conversation. erc proves useful in applications such as opinion mining and empathetic dialog systems. however, many of the existing models and datasets for emotion recognition are single-language. but, proliferating mixed language interactions have boosted interest in code-mixed natural language processing (nlp) tasks. the present work describes the system that participated in the shared task "emotion discovery and reasoning its flip in conversation (ediref)", task 10, organized at semeval 2024 @xcite . the ediref shared task is made up of three subtasks: (i) emotion recognition in conversation (erc) in hindi-english code-mixed conversations, (ii) emotion flip reasoning (efr) in hindi-english code-mixed conversations, and (iii) efr in english conversations. out of these subtasks, our team participated only in sub-task (i). many current approaches for diverse nlp tasks, including erc, relies on the application of large language models (llms) and fine-tuning them on a specific dataset. for this work we were interested in determining how existing language resources, such as emotion lexicons, could be used to complement and improve the predictions of llm-based approaches. for this reason, our final system, as detailed in section 4.2, is an ensemble of a bertbased implementation and traditional feature-based approaches, employing an emotion lexicon. apart from the emotion lexicon, we did not use any external datasets. only the dataset provided by the task organisers was used as an emotion annotated dataset. we also took into account the requirement expressed by the task organizers, that no data from task 2 or task 3 can be used to train/evaluate task 1. this paper is organized as follows: section 2 provides related work, section 3 briefly presents the task and describes the dataset, section 4 gives an overview of the participating system, including pre-processing and architecture, section 5 presents the results, and section 6 gives conclusions. de bruyne et al. (2022) evaluate the languagedependence of an mbert-based emotion detection model. experiments included the hindi and english languages. their findings suggest that there could be evidence for the language-dependence of emotion detection performance. datasets and systems for emotion recognition have been proposed for other languages as well. for example, for the romanian language, ciobotaru and dinu (2021) introduced the red dataset for emotion detection in romanian tweets. @xcite showed that particular romanian language words, such as negations, intensifiers and diminishers, affect the detected polarity of the sentiments described in natural language texts. furthermore, @xcite introduced robertweet, a bert-like llm for romanian language. the authors also describe a system using the robertweet model for emotion detection outperforming previous general-domain romanian and multilingual language models. laki and yang (2023) explore sentiment analysis with neural models for the hungarian language. the authors try to solve the class imbalance problem either by removing examples from the highly represented class (while keeping the same number of examples as the least represented class) or by duplicating examples from the least represented class. in addition they explore data augmentation by means of machine translation and cross-lingual transfer. different hungarian language llms, especially bert-like llms, are considered for the experiments. @xcite introduce hunembert, a fine-tuned bert-like model for classifying sentiment and emotion in political communication in the hungarian language. apart from neural network models and datasets, lexicons constitute another type of useful resources for sentiment analysis. this type of resources have been created for different languages. @xcite introduced the romanian emotion lexicon (roemolex v.3). it contains associations between a series of words and eight basic emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, trust) and two sentiment orientations (positivity and negativity). initially translated from an english version, it now contains additional tags, including derived emotions, part-of-speech, additional polarity scores and conceptual category information. it was also expanded with synonyms of the original terms and new words and phrases. @xcite propose the nrc word-emotion association lexicon (emolex). it contains english words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive). the annotations were manually done by crowdsourcing. the authors assess that despite some cultural differences, the majority of the affective norms are stable across languages. thus, the lexicon is also provided in over 100 languages by automatic translating the english terms using google translate. various datasets for sentiment classification, including those mentioned in this section, suffer from a class imbalance problem. frameworks for data augmentation, such as nl-augmenter @xcite , have been proposed, allowing automatic enrichment of less represented classes. @xcite proposed smote, a synthetic minority over-sampling technique. their approach combines under-sampling of the majority class with a special form of over-sampling the minority class. the minority class is over-sampled by creating "synthetic" examples rather than by over-sampling with replacement, thus reducing the potential overfitting. ## dataset and task the goal of the emotion recognition task is to classify a given sentence from a dialogue into one of eight emotion states: the seven universal human emotions as described by dr. paul ekman @xcite ("anger", "surprise", "contempt", "disgust", "fear", "joy", "sadness") and "neutral". the dataset files, with splits for training, validation, and testing, were provided in json format. the records contain fields for the name of the episode the lines were taken from, a list of speakers, the actual dialogue (list of sentences called "utterances"), and a list with the emotions attributed to each line ("emotions" or "labels"). the utterances included some unrecognized characters that needed to be removed. the training dataset contains 343 entries (8,506 utterances), the validation dataset contains 46 entries (1,354 utterances), while the test dataset contains 57 entries (1,580 utterances). the labels distribution for the train, validation and test splits is given in figure 1 . similar to other emotion recognition datasets, such as iemocap or meld, as reported in section 2, there is a class imbalance present in the task dataset as well. many sentences are marked as being "neutral", while the next class, considering the number of samples, is "joy". the least represented class is "disgust". ## results and discussion results are given in the proposed system tried to combine a lexicon approach with a llm prediction, considering that a manually created emotion lexicon could complement the llm predictions. nevertheless, even though the precision given by the random forest classifier based on features derived using the lexicon is surprisingly good, the recall is significantly lower, thus resulting in an overall lower f1 score, even in the face of a llm with a reduced number of parameters. in accordance with open science principles, the code for the described system is made available open source in its own github repository foot_2 . the class imbalance problem was tackled only with the smote technique. however, as mentioned in section 2, different frameworks for data augmentation are available. future work may include experiments with other data augmentation techniques for the minority classes. as documented in section 2, different authors have shown improvements using language-specific and domain-specific llms. for this work, we focused on a single bert llm. other llms, with more specificity or a larger number of parameters, may provide better results. however, the question regarding the possible enhancement of predictions using additional resources, such as emotion lexicons, still remains valid.