Sentiment Classification of Code-Mixed Tweets using Bi-Directional RNN and Language Tags (2021)

Abstract: Sentiment analysis tools and models have been developed extensively throughout the years, for European languages. In contrast, similar tools for Indian Languages are scarce. This is because, state-of-the-art pre-processing tools like POS tagger, shallow parsers, etc., are not readily available for Indian languages. Although, such working tools for Indian languages, like Hindi and Bengali, that are spoken by the majority of the population, are available, finding the same for less spoken languages like, Tamil, Telugu, and Malayalam, is difficult. Moreover, due to the advent of social media, the multi-lingual population of India, who are comfortable with both English ad their regional language, prefer to communicate by mixing both languages. This gives rise to massive code-mixed content and automatically annotating them with their respective sentiment labels becomes a challenging task. In this work, we take up a similar challenge of developing a sentiment analysis model that can work with English-Tamil code-mixed data. The proposed work tries to solve this by using bi-directional LSTMs along with language tagging. Other traditional methods, based on classical machine learning algorithms have also been discussed in the literature, and they also act as the baseline systems to which we will compare our Neural Network based model. The performance of the developed algorithm, based on Neural Network architecture, garnered precision, recall, and F1 scores of 0.59, 0.66, and 0.58 respectively.

## introduction sentiment analysis is the interpretation and classification of emotions (positive, negative, and neutral) within text data using text analysis techniques. it is one of the most important research areas in the domain of natural language processing (nlp) and has garnered much attention in the recent past. throughout the years, multiple state-of-the-art sentiment analysis models have been developed for the well known european languages, using classical machine learning (ml) algorithms as well as the recently developed neural network (nn) models. in contrast, very few such models have been developed for indian languages, due to their lower digital footprint, which results in the lack of annotated data. also, various pre-processing tools like parts-of-speech (pos) taggers, tokenizers, parsers, etc., for indian languages, are not readily available or are not of competitive quality. although, recent advances have been made for the indian languages that are spoken by the majority of the population, like hindi and bengali, the same cannot be said for under-resourced languages such as, tamil, telugu, and malayalam. for over 2600 years, recorded tamil literature has been documented. sangam literature, the earliest period of tamil literature, is dated from around 600 bc-300 ad. among the dravidian languages, tamil has the oldest existing literature. tamil is the oldest living language in india. moreover, with the advent of social media, sentiment analysis research has become even more wide-spread @xcite as it takes into account conversations of customers around the social space and puts them into context. but, in the context of the indian subcontinent, social media texts are not in one language and are largely code-mixed in nature. this is because india, has had much foreign acquaintance historically, and this has led the diaspora to adopt english as one of their official languages. due to this, much of the indian population are familiar with english as well as one or more regional languages @xcite . this leads to communication in sentences, which contain more than one language in the same phrase @xcite . furthermore, in a code-mixed communication, words of different languages are generally written in roman script, which leads to the formation of complex syntactical structures that are difficult to parse with traditional nlp tools. while traditional sentiment analysis models can model themselves on social media texts in one language, the same cannot be said for texts that are code-mixed in nature and also comprised of indian low-resourced languages. the proposed approach aims to mitigate this research problem for english-tamil code-mixed texts and uses bi-directional long-short-term-memory (lstm)s @xcite to tag the texts with their respective sentiment. language tagging of individual words was used as additional features while training the classification model. moreover, the training corpus was passed through fasttext @xcite embedding, to map the semantically similar words in a common 3d space. this mapping was also used to build the classification system. the designed model, when evaluated on test data, garnered an f1 score of 0.58. other baseline sentiment analysis models were also developed using classical ml algorithms and were used to compare the quality of the proposed algorithm developed by using nns. the rest of the paper is organized as follows. section 2 describes some of the previous research work conducted on the domain of language identification and sentiment analysis of code-mixed texts. section 3 describes the training and the test data used to develop and analyze our model. section 4 introduces the model developed for identifying languages of individual words in a code-mixed sentence. also, it describes our developed model and all the baseline models that were developed using traditional ml algorithms.finally, section 5 and 6 deals with the evaluation of our model and the concluding remarks. ## related work social media has become the voice of many people over the decades and it has special relations with real-time events. with its rise, a lot of data is being generated every day and information extraction from such data has become an important research area. also, the multi-lingual speaker, who prefers to communicate in more than one language, when expressing their opinions, generates a new kind of language, known as code-mixed language. since, these kinds of data are more or less always written in the roman script, analyzing these kinds of data, with help of nlp tools, becomes even more difficult. over the years, many experiments have been performed on code-mixed data. these include language identification, sentiment analysis, etc., to name a few. language identification tasks have been earlier performed on various language pairs, such as spanish-english @xcite , french-english @xcite , hindi-english @xcite and bengali-english @xcite . while these experiments were conducted with the help of dictionary word matching and ml-based algorithms such as support vector machines (svm), word-based logistic regression classifiers, and latent direchlet allocation (lda) @xcite , we use more state-of-the-art deep learning approaches to achieve the same. also, sentiment analysis or opinion mining from code-mixed data is a trivial task because • generally, code-mixed data is noisy in nature and requires cleaning and normalization. sentiment analysis of hindi-english code-mixed was performed by @xcite which used sub-word level representations in lstm architecture to perform it. shalini et. al. @xcite , attempted a case-study on sentiment analysis of english-kannada, english-hindi, and english-bengali texts using various machine and deep learning methods settings, like i. doc2vec+svm, ii. fasttext+softmax, iii. bi-lstm+softmax and iv. cnn+softmax. their reported results showed better accuracy when using deep learning methods as compared to traditional machine learning methods. our work, on the other hand, is an amalgamation of all the methods pointed out earlier and incorporates language identification modules, the usage of fasttext embeddings, and bi-lstm cells to develop the deep learning model. ## data the data for building the sentiment analysis model for english-tamil code-mixed data was collected from the "dravidian-codemix -fire 2020" foot_0 shared task. the organizers of the task provided us with tamil-english and malayalam-english codemixed text data, derived from youtube video comments. the dataset contained all the three types of code-mixed sentences -inter-sentential switch, intra-sentential switch, and tag switching and had five output labels; positive, negative, mixed feelings, not tamil, and unknown state. most comments were written in roman script with either tamil / malayalam grammar with english lexicon or english grammar with tamil / malayalam lexicon. some comments were written in tamil / malayalam script with english expressions in between. further, the english-tamil dataset was divided into training, validation, and test data which had 11,335, 1,260, and 3,149 code-mixed sentence instances respectively. after the pre-processing step, we proceeded with tokenizing the cleaned sentences using the nltk 2 library. subsequently, we used this data to train fasttext embedding. this was done, to map the words with similar meaning and context, close to each other in a 3d space. the skip-gram model was used instead of the continuous-bag-of-words (cbow) model as skip-gram works best for low data sizes. the model took into account character n-grams from 3 to 6 characters. using the trained model, we were able to extract word vectors of size 100. these word vectors were preserved to be used as input for our sentiment analysis model. ## evaluation from tables 2 and 3 , we can see that though the ml and dl models perform neck-in-neck, but still, we preferred the dl model, developed using bidirectional lstm's and language tag feature as it garnered the highest f1-score. this model was then tested using 3,149 test data, provided by the shared task organizers. the results of the testing phase of the selected model are quantified in ## conclusion in the current work, we attempted to solve the problem of sentiment analysis of code-mixed english-tamil sentences. our system was based on using bi-directional lstm along with language tag features. also, fasttext embedding was used to generate word vectors to train the model. for predicting the language tags, another deep learning system, based on character embedding was also developed. other models, based on traditional ml algorithms were also developed that was used to compare our developed model. our system, when evaluated on the test data, garnered an f1 score of 0.58. as future work, we would like to increase this data, as deep learning algorithms tend to work well with higher amount of data and use state-ofthe-art neural network architectures, like bert, roberta, etc., on this data, taking into advantage the concept of matrix and embedded language, sen-tiwordnet, and other nlp features.