IRL ab_ DAIICT at S em E val-2020 Task 9: Machine Learning and Deep Learning Methods for Sentiment Analysis of Code-Mixed Tweets (2020)

Abstract: The paper describes systems that our team IRLab_DAIICT employed for the shared task Sentiment Analysis for Code-Mixed Social Media Text in SemEval 2020. We conducted our experiments on a Hindi-English CodeMixed Tweet dataset which was annotated with sentiment labels. F1-score was the official evaluation metric and our best approach, an ensemble of Logistic Regression, Random Forest and BERT, achieved an F1-score of 0.693.

## introduction the past few years has seen a massive surge in the number of people using social media sites such twitter, facebook, etc., in the indian subcontinent. a large faction of these netizens do not always use unicode to voice their opinions on social media platforms. instead, they resort to roman script/transliteration and frequent insertion of english words or phrases through code-mixing, with many using a mix of multiple languages. this points towards a growing need for technologies that can handle such code-mixed text, especially in india which is home to a plethora of languages. sentiment analysis being a core nlp feature, especially for social media analysis, makes the development of systems capable of performing accurate sentiment analysis on code-mixed data an immediate necessity. the sentmix @xcite task was organised for hindi-english and spanish-english codemixed data. the data comprised of tweets which had been annotated with token-level language labels and sentence-level sentiment labels. our team conducted experiments on the hindi-english dataset. our experiments followed two approaches, the first involved feature extraction using tf-idf and using logistic regression or random forests to classify tweets as positive, negative or neutral. the second involved transfer learning using bert. we tried the this approach because bert utilizes a sub-word vocabulary and hence, can handle oov words allowing us to use a bert @xcite pre-trained on english data for hindi-english code-mixed data. the rest of paper is organized as follows. in section 2 we discuss related work. in section 3 we briefly discuss the dataset which was provided. proposed methods and obtained results are presented in section 4. finally we conclude our work and future works in section 5. ## related work given the rise to prominence of social media platforms like facebook and twitter in the past few years, sentiment analysis of social media posts has become a popular avenue for research. sentiment analysis of code-mixed data is considered a difficult task as the data is quite noisy and there is a lack of large volumes of annotated data. a popular shared task which focuses on sentiment analysis of tweets is the sentiment analysis for indian languages (sail). sail 2015 focused on performing sentiment analysis on tweets in three indian languages: bengali, hindi, and tamil @xcite . sail-2017 had two code-mixed datasets hindi-english and bengali-english for developing sentiment analysis systems @xcite . k. @xcite combined tf-idf with gain ratio based feature selection and rbf neural network for sentiment analysis on hindi-english code-mixed data. @xcite generated sub-word representations of hindi-english code-mixed data collected from facebook using character embeddings and 1-d convolution. the sub-word representations were then passed to lstm layers for sentiment classification. k. @xcite investigated the application of transfer learning for hate speech detection on code-mixed data. they trained cnn-based neural models on tweets in a chosen primary language, followed by fine-tuning on a smaller dataset which had been transliterated to the primary language. ## dataset the organizers provided a dataset containing hindi-english code-mixed tweets. the dataset had a few inconsistencies with respect to annotation. in certain cases the lang id given for a token was incorrect. emoticons were tagged as 'o' and in some cases and as 'emt'. this prompted us to not use the lang id annotation in our experiments. some of the tweets were not english-hindi code-mixed. non-hinglish tweet (meta:4330): "nen á vist bolest vztek smutek zmatek osam ě lost beznad ě j a nakonec jen klid asi takhle vypad á m ů j life ..." tokens nen á vist bolest vztek smutek zmatek osam ě lost beznad ě j a nakonec jen klid asi takhle vypad á m ů j life ... lang id eng o eng eng eng eng hin hin o eng eng o hin eng eng hin hin hin hin hin o hin o eng eng o wrong language id annotation (meta:15893): "evryone shit the fuck up https//t.co/rin0mqkpsz" tokens evryone shit the fuck up https // t . co / rinomqkpsz lang id hin hin hin hin eng eng o eng o eng o hin different tags used for tagging emojis (meta:11381): "rt @ambar aum @shoaib100mph na naa your boys played really well! this meme explains it all #pakvwi #wivpak https//t.co/rcoklf01ny" tokens rt @ ambar aum @ shoaib100mph na naa your boys played really well ! this meme explains it all ... lang id eng o hin o eng o eng eng hin hin eng eng eng eng o o eng eng eng eng eng emt ... ## conclusion in this paper we presented methods used to solve the sentmix task organised by semeval 2020 and our results for the task. we utilized feature extraction using tf-idf and machine learning techniques like logistic regression/ random forest but these methods were outperformed by our transfer learning approach which used an english language bert. a possible cause can be the vocabulary of the corpus. it is quite common for hindi-english code-mixed data to have different spellings (tweets in general contain a lot of spelling errors and internet jargon) for the same hindi word which leads to a higher frequency of out-of-vocabulary (oov) words. since transformer-based language models, like bert, employ subword embeddings these oov words can be handled without much overhead whereas the same is not possible for tf-idf. in future we would like to improve our results by including emoticons and employing code-mixed data specific preprocessing techniques.