CLT eam1 at S em E val-2024 Task 10: Large Language Model based ensemble for Emotion Detection in H inglish (2024)

Abstract: This paper outlines our approach for the ERC subtask of the SemEval 2024 EdiREF Shared Task. In this sub-task, an emotion had to be assigned to an utterance which was the part of a dialogue. The utterance had to be classified into one of the following classes- disgust, contempt, anger, neutral, joy, sadness, fear, surprise. Our proposed system makes use of an ensemble of language specific RoBERTA and BERT models to tackle the problem. A weighted F1-score of 44% was achieved by our system in this task. We conducted comprehensive ablations and suggested directions of future work. Our codebase is available publicly.

## introduction language has been the primary mode of communication for humans since pre-historic times. in linguistics, code-mixing traditionally refers to the embedding of words or phrases into an utterance of another language @xcite . in many multi-lingual societies we see the development of code-mixed languages. hinglish is one such language which is a linguistic blend of hindi and english which is spoken primarily in india. hinglish generally refers to hindi that is written in the roman script and is used in combination with some english phrases. the variance in spellings and the multiple interpretations of hindi words, depending on specific contexts, pose challenges for the analysis of language. the semeval workshop (co-located with naacl 2024) explores and advances the current state of semantic analysis to tackle increasingly complex problems in natural language semanitcs. this paper outlines our approach for the emotion recognition in coversation (erc) @xcite sub-task of the emotion in this sub-task we had to assign a specific emotion to an utterance which the part of a dialogue. each episode had multiple speakers speaking in hinglish. we ranked 11 th in this subtask achieving a weighted f1-score of 44%. an end-to-end deep learning pipeline that uses an ensemble of transformer-based hinglish models was used. we converged on the best models to use in the ensemble by rigorous experimentation using the available models. we also analyse the performance of the classification pipeline and present ablations. we also elaborate on the shortcomings of our systems and some future directions of work. for ensemble learning, @xcite have proposed an ensemble model which generates predictions after the text passes through a vectorisation layer having 2 outputs, one of which is represented as a bag-of-words model and provided as input to 3 voters, namely naive bayes (nb), support vector machine (svm) and decision tree (dt); and another is a direct input to a cnn. @xcite proposes a new sentiment analysis method, based on text-based hidden markov models, that uses word orders without the need of sentiment lexicons. @xcite proposed use of ensemble feature selection for multi-label text classification which has been used in our approach. ## data the data is in the hindi-english (hinglish) codemixed format which contains words spoken in hindi but written in the roman script and english words. the dataset consisted of 343 episodes or dialogues and contained a total of 8,506 utterances which had to classified into eight classes -disgust, contempt, anger, neutral, joy, sadness, fear, surprise. the validation dataset consisted of 46 episodes having 1,354 utterances. the system was then evaluated on a test dataset that contained 57 dialogues consisting of 1,580 utterances. we have illustrated the data distribution in the training dataset in figure 1 . there is acute class imbalance. the class "neutral" contains the most samples (3,123) while the class "disgust" contains the least samples (103). the imbalance ratio was almost 1:30. to mitigate this we tried oversampling to increase size of examples for classes having lower utterances, but they did not improve the performance of the system. ## system description the chosen sub-task of emotion detection was a multi-class classification problem which required an utterance to be classified into one of 8 classes. we performed basic pre-processing on the text before passing it to the model. this includes removal of stopwords and punctuation marks from the text, as well as spelling normalisation from the dataset. due to scarcity of domain specific data related to this task we decided to fine-tune existing transformer-based models to adapt them for our task. models from (nayak and joshi, 2022) like hingbert, hingroberta, hingmbert which are based on bert and roberta @xcite that were pre-trained on hinglish data scraped from twitter were chosen for the task with multilingual models like m-bert @xcite . we also chose a variant of bert (hinglish-bert) 2 and a hingbert variant that was fine-tuned on on the l3cube-hinglid @xcite corpus to include in our system. a linear layer was connected to the pooler output of these models and they were fine-tuned on the dataset. we observed that the performance of the system was enhanced when an ensemble of models was used. we use the method of hard voting to obtain the results from the ensemble. if there was no consensus reached in the ensemble, then the label that the model with the highest f1-score predicted was used as the prediction of the system. ## conclusion this paper aims to describe our approach for the erc sub-task of the 2024 ediref shared task. we conducted experiments with multiple transformer based models like hingbert, hingbert and mbert. we also show that an ensemble of these models has the best performance on the evaluation dataset with a weighted f1-score of 44%. we foresee several future directions. one direction can be to develop and use more sophisticated methods for ensembling. another direction is the generation or collection of such data which is more relevant in a real-world scenario in low-resource languages.