TECHSSN at S em E val-2024 Task 10: LSTM -based Approach for Emotion Detection in Multilingual Code-Mixed Conversations (2024)

Abstract: Emotion Recognition in Conversation (ERC) in the context of code-mixed Hindi-English interactions is a subtask addressed in SemEval-2024 as Task 10. We made our maiden attempt to solve the problem using natural language processing, machine learning and deep learning techniques, that perform well in properly assigning emotions to individual utterances from a predefined collection. The use of well-proven classifier such as Long Short Term Memory networks improve the model’s efficacy than the BERT and Glove based models. How-ever, difficulties develop in the subtle arena of emotion-flip reasoning in multi-party discussions, emphasizing the importance of specialized methodologies. Our findings shed light on the intricacies of emotion dynamics in code-mixed languages, pointing to potential areas for further research and refinement in multilingual understanding.

## introduction the ultimate objective of this task ediref is to make progress in the field of conversational emotion recognition and reasoning. analyzing emotions in natural language offers measurable understandings within the typically subjective domain of expressive language, connecting disciplines like psychology, cognition, and linguistics. this focuses on code-mixed hindi-english dialogues, creating a unique and challenging linguistic setting in which participants must decipher the complexities of emotion recognition (erc) and emotion flip reasoning (efr). code-mixing, or the intentional use of different languages within a single conversation, complicates the process and necessitates novel techniques for effective emotion recognition. this kind of collaborative work is critical because it reflects the changing environment of communication, where diversified language use needs complex models capable of understanding emotions in multilingual discussions resulting in overall development. our approach which focuses on the emotion recognition in conversation (erc) subtask, employs a sophisticated strategy based on deep learning methodologies suited to the intricacies of codemixed hindi-english (hi-en) talks. at its core, our solution employs a bidirectional long short-term memory (bi-lstm) network, an architecture capable of capturing intricate sequential connections inside text. we chose lstm as lstm networks are designed to overcome the limitations of traditional recurrent neural networks (rnns) by mitigating the vanishing gradient problem and lstm's ability to retain and selectively update information over time can lead to more accurate predictions of emotional states. we prioritized preprocessing, which includes tokenization and sequence padding, to help the model understand speech contexts. to enhance linguistic representation, the embedding layer is initialized with pre-trained word embeddings. for regularization and addressing overfitting issues, strategically placed dropout layers are incorporated. the training process utilizes categorical cross-entropy loss and the adam optimizer. the model showcased proficiency in precisely attributing emotions to individual utterances, showcasing its capability to decipher intricate emotional expressions in a multilingual context. from a quantitative standpoint, our system achieved an accuracy of 0.378 in sentiment analysis and secured the 23rd position among the competing teams. these findings not only provide insights into the model's strengths and areas for improvement but also highlight the importance of specialized mechanisms for complicated emotional connections in multilingual communication such as expanding datasets to encompass a broader range of linguistic and cultural contexts, as well as areas for future research 1 763 2 background the task at hand focuses on comprehending and categorizing emotions presented in code-mixed hindi-english interactions. in this context, the input comprises dialogues where individuals communicate in both hindi and english. the primary aim is to analyze each segment of these interactions and assign a precise emotion from a predefined set to capture nuanced sentiments. for example, throughout a conversation, people may display a variety of emotions at different points, such as joy, sorrow, or rage. the task comprises two datasets centered around multi-modal sarcasm detection and humor classification (masac) for sub-tasks 1 and 2, and multimodal emotion lines (meld) for subtask 3, but this paper majorly focuses on sub-task 1. the datasets for this task consist of code-mixed conversations, reflecting the real-world scenario where individuals seamlessly blend hindi and english while communicating. the genre of the conversations may vary, encompassing diverse topics and contexts to ensure a comprehensive understanding of emotion dynamics in code-mixed language interactions. the size of the datasets is not explicitly mentioned, but it likely involves a substantial amount of annotated dialogues to train and evaluate emotion recognition models effectively. the dataset consisted of four parameters the episode name, the speakers list, the utterances, and the emotions mapped to the respective utterances. the emotions mapped to the utterances provided a standard for assessing models' performance in recognizing emotional expressions. for example, the utterance "ok, chalo rosesh chalo bahar" was mapped to the emotion "contempt". ## related work arora et al. @xcite explores the capabilities of deep neural networks (dnn) using rectified linear units (relu). it introduces an algorithm for training a relu dnn with one hidden layer to global optimality with polynomial runtime in data size. the paper also improves lower bounds for approximating relu deep net functions and provides gap theorems for smoothly parametrized families of "hard" functions. notably, it demonstrates the existence of functions requiring k 3 total nodes in a relu dnn with k 2 hidden layers, shedding light on the network's complexity. anshul wadhawan and akshita aggarwal et al. @xcite presents a transformer-based approach for detecting emotions in code-mixed tweets. they introduce a hinglish dataset, use bilingual word embeddings, and experiment with various models, including cnns, lstms, and transformers like bert. the bert model achieves the best accuracy at 71.43%. the paper highlights the importance of emotion detection in social media and multilingual contexts, providing a valuable annotated dataset for future research. shivani kumar et al. @xcite delved into emotion flip reasoning (efr) in multiparty conversations, showcasing state-of-the-art performance against baselines. their research highlights the significance of efr in enhancing empathetic response generation and understanding emotional dynamics in conversational settings, thus addressing the gap and providing insights into how specific remarks or expressions affect listeners. deepanshu vijay et al. @xcite addresses emotion prediction in hindi-english codemixed social media text. they introduce a corpus from twitter annotated with emotions and source languages. the paper proposes a supervised classification system using machine learning techniques and diverse features for emotion detection, contributing to resources for hindi-english code-mixed text analysis in multilingual contexts. in a parallel domain, a study focused on emotion analysis in low resource language tamil is done by @xcite . they have employed a lexicon-based approach and transformer models, utilizing dictionaries of words labeled with emotions. this research specifically addresses the challenges of extracting emotions from low resource texts in social media contexts, offering valuable insights. contextual emotion detection is executed using gaussian model and ensemble model by @xcite . the challenge of contextual emotion detection in natural language processing has been addressed, emphasizing the difficulty for both machines and humans to accurately detect emotions like sadness or disgust in a sentence without sufficient context. the study underscores the growing importance of providing sensible responses in text messaging applications, where digital agents play a prominent role. the research showcases the efficacy of a gaussian process for detecting contextual emotions within sentences, comparing its performance with decision tree and ensemble models, including random forest, adaboost, and gradient boost. emotion recognition in hindi-english codemixed data, as explored in relevant papers, employs models like bert, roberta, cnns, and lstms. the challenges highlighted align with our work, emphasizing the importance of addressing code-mixing complexities and the scarcity of annotated datasets. similarly, in corpus creation for emotion prediction, the focus on a twitter-based annotated corpus resonates with our efforts. the shared emphasis on overcoming linguistic diversity and cultural nuances underscores the mutual pursuit of enhanced accuracy in emotion recognition, urging continued research in these aspects. ## system overview to optimise efficiency, we methodically integrated numerous critical algorithms and modelling decisions into our sentiment analysis model. ## experimental setup the data split used in the given code divides the dataset into training and testing sets. the split ratio is 80% for training data and 20% for testing data, as stated by the code's test_size argument of 0.2 to prevent overfitting and perform better on new data. the learning rate is a crucial hyperparameter that determines the step size during the optimization 4 766 process.the learning rate, a key factor in the optimization process, was explored with values such as 0.01, 0.001, and 0.0001. to mitigate overfitting, dropout rates were varied, including options like 0.2 and 0.5 the number of lstm units in the bidirectional lstm layer, a crucial aspect of model capacity, was adjusted with values like 32, 64, and 128. additionally, the impact of different batch sizes (16, 32, 64) on convergence and computational efficiency was systematically explored. in our experimental setup, we harnessed the power of scikit-learn for seamless implementation of various machine learning algorithms, handling data preprocessing tasks, and evaluating performance using diverse metrics. the nltk library, an essential component, efficiently managed critical natural language processing functions, including tokenization, stopwords removal, and stemming. to ensure model persistence and flexibility, we adopted joblib, a tool adept at saving and loading trained models. moreover, our approach integrated external tools like nltk indic nlp, scikit-learn, and tensorflow to elevate specific components of our sentiment analysis model. this encompassed optimizing tokenization, refining data splitting techniques, streamlining preprocessing steps, and conducting rigorous model evaluations, all contributing to the robustness and effectiveness of our experimental framework. ## results the task is evaluated using the following performance metrics: precision, recall, accuracy and f1score. recall indicates the classifier's ability to identify positive instances accurately and accuracy is defined as the ratio of the correctly predicted instances to the total number of instances in a dataset. it acts as a straightforward for the model's correctness while precision is a measure of how accurate the positive predictions are. we used a variety of approaches to improve the accuracy of sentiment analysis. one important approach was using pre-trained word embeddings, such as word glove @xcite , which captures semantic associations between words. the use of word glove embeddings enabled a decent comprehension of contextual nuances, which contributed to increased sentiment analysis results with a f1 score of about 0.35. while we initially explored the use of word glove embeddings, we found that other methodologies yielded better results for our specific task. therefore, we transitioned away from word glove embeddings and pursued alternative approaches that demonstrated improved performance in managing the challenges associated with dual tokenization in mixed-language conversations. we also attempted to develop dist-bert, a transformer model, but faced a difficult case in which its integration resulted in an underwhelming f1 score of 0.30. this unexpected outcome spurred a rethinking of the implementation, prompting us to investigate alternate tactics and optimisations to improve the model's effectiveness. owing to the model's training on the restricted quantity of available data, it exhibits a bias towards specific emotions. "okay chaliye dad, mein aapko bahar fenk kar aata hun!" is an example of a statement that should be predicted as "joy," instead it is predicted as "neutral." the reason for the model's behaviour is that a lot of utterances are mapped to the neutral emotion; as a result, when a model is trained on this kind of data, it naturally becomes biased towards such emotion types. in addition, a thorough analysis of the classification reports shed additional light on the theory on how class imbalances affect the model's functionality. as can be seen from the performance measures that were previously addressed, there are significant differences in the weighted and macro f1-scores, even if the classifiers' accuracy is the same for all datasets that were used. in particular, the sentiment analysis model performs noticeably better on the hindi-english code-mixed dataset than on the hindi and english monolingual datasets, highlighting the difficulties caused by the imbalances in the latter. this limitation is particularly notable for emotions with limited representation in the training data, emphasizing the need for strategies to address imbalances and improve the models' robustness across diverse linguistic scenarios. ## conclusion we lay out a sentiment analysis system that can handle hindi-english talks with mixed codes. recognising and rationalising emotions in a bilingual environment was the main goal. the results show that the participants performed competitively in terms of emotion perception and reasoning, especially when there was frequent language change. the model using lstm layers, nltk indic nlp provided the best result of 0.36 f1 score. subtle emotional cues and particular code-mixing patterns continue to provide difficulties, nevertheless. our method is noteworthy for its hybrid approach, which makes use of sentiment analysis, contextual embedding techniques, and language models that have already been trained and refined using codemixed datasets. it is still imperative to address specifics in lowresource languages in future development. techniques like compiling lists of language-specific stop words have shown to be effective. moreover, the effect of class disparities on the model's functionality is recognized. furthermore, the effect of class imbalances on the performance of the model is recognized. subsequent research could investigate customized approaches, including data enrichment or clustering techniques, to address these imbalances and improve the model's flexibility in a variety of language circumstances unique to our code.