Hidetsune at S em E val-2024 Task 10: An E nglish Based Approach to Emotion Recognition in H indi- E nglish code-mixed Conversations Using Machine Learning and Machine Translation (2024)

Abstract: In this system paper for SemEval-2024 Task10 subtask 1 (ERC), I present my approach to recognizing emotions in Hindi-English codemixed conversations. I train a SpaCy model with English translated data and classify emotions behind Hindi-English code-mixed utterances by using the model and translating them into English. I use machine translation to translate all the data in Hindi-English mixed language into English due to an easy access to existing data for emotion recognition in English. Some additional data in English are used to enhance my model. This English based approach demonstrates a fundamental possibility and potential of simplifying code-mixed language into one major language for emotion recognition.

## introduction semeval 2024 task 10 @xcite calls for assigning emotions to hindi-english codemixed conversations (erc) and reasoning emotion flips (efr) in hindi-english code-mixed conversations and in english conversations. i participate in subtask 1 (erc), for which hindi-english codemixed utterances are given in text for participants to recognize emotions behind them. an emotion from disgust, contempt, anger, neutral, joy, sadness, fear and surprise is assigned to each utterance as the correct emotion associated with it. my methodology has its basis on emotion detection in english. i translate all the development data into english using machine translation @xcite , and use the data to train a spacy model @xcite . i also use data by @xcite to enhance my model, where utterances in english and their previously assigned emotions are given as a csv file. he mainly introduces his own model, but i made use of the data only. i leverage an easy access to emotion-assigned data in english and its linguistic simplicity. therefore, the point of my approach is to test whether or not machine translation of hindi-english codemixed conversations into english can contribute to fundamental accuracy for emotion recognition with lower complexity and an easier access to additional data. the result shows that combination of machine translation and machine learning works with reasonable accuracy for emotion detection considering its simplicity and numerous data in english that can be implemented in future studies. my code is available on github foot_0 . ## background the subtask i participate in focuses on emotion recognition in hindi-english code-mixed conversations. given hindi-english code-mixed utterances ("kuchh karo sahil please kuchh karo. mera rosesh adopt ho karke chala gaya na to me, i know this sounds horribly melodramatic, monishaish, par me mar jaaungi. i swear mein mar jaaungi" for example) as input, the subtask requires assigning an emotion to each of them as output. episode name and speakers' names are given as input as well, but they are not used for my solutions. before proceeding with this task, hindi-english code-mixed language needs to be explained in detail. hindi-english code-mixed language, which is often referred to as hinglish, is a language in which speakers mix hindi and english in conversations. according to a study by @xcite , there are some hinglish speakers who cannot speak pure hindi, and even those who speak both hindi and hinglish tend to speak hinglish in conversations with monolingual speakers of hinglish, causing the number of monolingual speakers of hinglish to grow as a result. therefore, the situation in this task is not a small topic but rather an essential one for the future indian communities. most previous studies aim to recognize emotions in hinglish by collecting hinglish sentences and using them directly for their approaches. a study by @xcite uses n-grams for their solution with hinglish on twitter. the work detects emotions with high accuracy, but they state that the accuracy drops by nearly 16% without char n-grams. another previous work by @xcite , which uses the solution by @xcite as their baseline, collects more data with 12000 hindi-english code-mixed sentences for training. in addition to these, a study by wadhawan and aggarwal (2021) achieves high accuracy with a transformer based bert model, and another study by @xcite deals with hinglish on youtube comment sections. in all of these works, they process hindi-english code-mixed sentences without translating them into other languages. on the other hand, one of the biggest issues with emotion recognition in hinglish conversations might be shortage of datasets. although there are many hinglish sentences expressing emotions on social media platforms, the language they use on them might differ from in real conversations to some extent. taking the availability of numerous existing datasets in english into account as well, i decided to explore a solution to use english translated data rather than the original hinglish data. this trial requires machine translation, and this step is combined with classical machine learning method, achieving reasonable accuracy for its simplicity with an additional english dataset concatenated to the translated dataset. emotion recognition in hinglish can be complicated because the language has technically two languages (hindi+english). however, when simplified into all in english properly, there is much more potential to deal with hinglish emotion recognition with an easy access to enormous english datasets and established nlp methods that have been mainly used for english. this paper aims to guide a direction to future application of this approach, establishing the basis with easily used datasets and model. ## system overview the main strategy of my system is a combination of classical machine learning method with simplification of hindi-english code-mixed sentences into english sentences. therefore, my methodology is composed of data preparation using machine trans-lation and the classic machine learning process with training. a quick overview of my algorithm is as follows. for data preparation, i use the official training and development dataset @xcite , and additional data by nikam (n.d.) to enhance my model and mitigate data imbalance. firstly, i import the official datasets in json file format @xcite , and convert them into pandas dataframes which have episode names, utterances in hindi-english mixed language (pronunciation forms of hindi + english), speaker names and emotions as columns. then, i use google translate @xcite to translate all the utterances in hindi-english mixed language into english, and i add the translated sentences to the dataframe as a new column named "utterances_english". since episode name, utterances in mixed language and speaker names are of no use at this time, they are dropped from the dataframe to have only "emotion" and "ut-terances_english" as columns on the dataframes. after that, they are concatenated into one dataframe. there is a huge data imbalance and shortage of training data at this point, so i use additional data by nikam (n.d.) to mitigate these problems. in the next step, i train the spacy-v3 model @xcite with the prepared data and use it to assign emotions to unlabeled evaluation data. the evaluation data are composed of episode name, utterances in hindi-english mixed language, speaker names as columns. "utterances_english" column, in which machine translated sentence is assigned to each given utterance, is added to have the model predict emotions behind the utterances. participating in this emotion recognition task using the combined strategy enables me to explore the potential of implementing machine translation into a specific situation as this task. the application of machine translation to machine learning makes it possible for a multi-label text classifier to predict emotions behind hinglish sentences with reasonable accuracy for the entry of this trial. considering the saved complicated steps needed to handle the data as text in two separate languages, you can see the potential of simplification that my methodology aims at by combing machine translation with machine learning. future applications for higher accuracy might include more training data in hindi-english mixed sentences and enhancement of the model with the implementation of machine translation into hindi written sentences. participating in this task reveals both strengths and weaknesses of my strategy, guiding directions of future studies to apply machine translation and classic nlp methods to emotion recognition in hinglish conversations. ## experimental setup before moving on to the actual training of the model, some setups were required to prepare training data. as stated before, the utterances in the provided development data @xcite are all in hindi-english mixed language. since my participation in this task intends to combine machine translation with machine learning for emotion recognition, i decided to simplify the data by translating them into one language rather than taking the complexity of separating them into the two languages. this simplification by machine translation might have changed the original meanings of the utterances, but the difference might not be significant for recognizing the emotions only. looking through the hindi-english code-mixed sentences, i noticed that fairly large parts of the utterances are in pronunciation forms in hindi and that english is used only partly. for instance, some short words or phrases like "goodbye! there could be two possible choices in my approach; translate all the mixed language sentences (hindi+english) into hindi or into english. it might be a good idea as well to choose the former considering the high prevalence of hindi, but i chose the latter due to the higher availability of reliable additional data on emotion recognition in english. as for machine translation, google translate (googletrans 3.1.0a-0) was used. the reference @xcite uses it to translate sentences in all actual hindi characters (no english and in written form of hindi; not in pronunciation form). on the other had, i undertook an experiment to try it for the official development dataset (hindi-english codemixed and pronunciation form for hindi; not in written form) and founded that it works. therefore, all the given utterances are translated into english and added to the dataset as a new column. at the next step, additional data are collected and processed to be concatenated with the translated version of official data. since there is a huge data imbalance and lack of training data as stated in the previous section, it is obvious that additional data is needed where most of the 8 emotions used in this task (disgust, contempt, anger, neutral, joy, sadness, fear and surprise) are labeled for utterances or sentences. for consistency of additional data, one source was used rather than concatenating multiple sources with different categories of emotions. data by nikam (nikam, n.d.) were chosen for additional data. the original concatenated data (the official training and validation data) are composed of text in english and 8 emotions (disgust, anger, neutral, joy, sadness, fear, contempt and surprise) for each utterance. the only two differences in categories of emotion are that the additional data does not have "contempt" while the official one have, and that the additional data have "shame" while the official one does not. details are as shown on table1. the concatenated version of official data and the additional data are concatenated separately for each emotion, and the number of utterances is adjusted after that. conducting some experimental trial on development phase, i decided to limit the data so that it has 3001 utterances (3000th rows counting from 0) at maximum for each emotion type, and all the utterances that exceed the number (from 3002th rows) are dropped from the training dataset. ## results in the evaluation phase, my trained model worked with accuracy of 0.39 (weighted f1 score calculated by the organizers' system ), and the ranking was 17th out of 39 participants. the result is not as good as to use for a practical use as of now, but it definitely shows a basic ability of my approach to apply machine translation to hindi-english codemixed language. there are certain weaknesses in my algorithm as the result shows. one of the biggest weaknesses is the lack of dataset due to inevitable data imbalance. concatenating the official data @xcite with additional data (nikam, n.d.), there are only 1004 utterances associated with disgust for example while there are many more utterances than 3001 for anger. limiting the number of utterances up to 1001 for each emotion yielded low accuracy of 0.27 in the development phase, so i chose to accept data imbalance to some extent with the limitation of 3001 utterances for each emotion so that the model is trained better. larger data imbalance was inevitable to make use of more data, yielding lower scores in development phase, so i had no choice but to decide on around that limitation. however, it cannot be said that less than around 3000 training utterances per emotion are enough for accurate emotion recognition, which is one of the main weaknesses of my solution in this task. in addition to that, the process of machine translation might have changed the original meaning of the utterances, which might have lowered the quality of training. i cannot look deeper into this possible issue because i am not a hindi speaker, but the accuracy might go up with a better machine translator. another weakness of my solution is that the additional data i used (nikam, n.d.) are provably not from actual conversations. there are many "@" followed by what i suppose are usernames, so the entire additional data (nikam, n.d.) are probably from social media platforms. since this task deals with text version of conversations, situations of the datasets are probably unmatched with each other. on the other hand, the strength of my approach is the linguistic simplicity that enables the model to have its potential to utilize established nlp techniques and numerous data that have been developed for english. since english is used widely around the world, there are tons of other data sources that can be implemented into my model. it goes without saying that there are already many existing data in which emotions are labeled with sentences, i can also write down daily conversations in english into text and label each utterance with an emotion for higher accuracy since this task deals with data from conversations. ## conclusions to summarize, i firstly simplified utterances in hindi-english code-mixed language by translating them into english. machine translation @xcite is used in this step, and additional data (nikam, n.d.) are concatenated to mitigate data imbalance and enhance the model. utterances of evaluation data @xcite are also translated into english by machine translation in the same way, and the trained model predicted an emotion for each utterance. although my solution has a room for improvement, the result shows a basic ability of the simplification by machine translation to recognize emotions behind hindi-english code-mixed utterances. given the abundance of data and established nlp techniques in english, my approach, combining machine translation with classical nlp methods, might open the door for addressing the challenges in emotion recognition in hinglish caused by its linguistic complexity.