Sentiment Analysis of D ravidian Code Mixed Data (2021)

Abstract: This paper presents the methodologies implemented while classifying Dravidian code-mixed comments according to their polarity. With datasets of code-mixed Tamil and Malayalam available, three methods are proposed - a sub-word level model, a word embedding based model and a machine learning based architecture. The sub-word and word embedding based models utilized Long Short Term Memory (LSTM) network along with language-specific preprocessing while the machine learning model used term frequency–inverse document frequency (TF-IDF) vectorization along with a Logistic Regression model. The sub-word level model was submitted to the the track ‘Sentiment Analysis for Dravidian Languages in Code-Mixed Text’ proposed by Forum of Information Retrieval Evaluation in 2020 (FIRE 2020). Although it received a rank of 5 and 12 for the Tamil and Malayalam tasks respectively in the FIRE 2020 track, this paper improves upon the results by a margin to attain final weighted F1-scores of 0.65 for the Tamil task and 0.68 for the Malayalam task. The former score is equivalent to that attained by the highest ranked team of the Tamil track.

## introduction code-mixing usually involves two languages to create another language that consists of elements of both in a structurally understandable manner. it has been noted that bilingual and multilingual societies use multiple languages together in informal speech and text. dravidian code-mixed languages, including but not limited to malayalam and tamil, are increasingly used by younger generations in advertising, entertainment and social media. the language is commonly written in roman script. with the rise in the number of non-english and multilingual speakers using social media, there is an interest in analysing the sentiment of the content posted by them. as code-mixed data does not belong to one language and is often written using roman script, identifying its polarity cannot be done using traditional sentiment analysis models @xcite @xcite . in social media, low-resourced languages such as tamil and malayalam have been increasingly used along with english @xcite . indian government declared tamil as classical language in 2004, meaning that it meets three criteria: its sources are old; it has an independent tradition; and it has a significant body of ancient literature. the most ancient non-sanskritic indian literature is found in tamil. the earliest existing tamil literary works and their commentaries commemorate the organisation of long-term tamil sangams (600 bce to 300 ce) by the pandiyan rulers, which studied, established and made improvements in the language of tamil. identifying the sentiment of this data can prove to be useful in social media monitoring and feedback of users towards other online content. the 'sentiment analysis for dravidian languages in code-mixed text' task proposed by fire 2020 @xcite ) contains a code-mixed dataset consisting of comments from social media websites for both tamil and malayalam. each team had to submit a set of predicted sentiments for the tamil-english and malayalam-english mixed test sets @xcite . along with language specific preprocessing techniques, the implemented model made use of sub-word level representations to incorporate features at the morpheme level, the smallest meaningful unit of any language. evaluated by weighted average f1-score, the subword level approach achieved the 5th highest score in the tamil task and the 12th rank in the malayalam task @xcite . this paper presents the aforementioned model and two more models -a word embedding based model and a machine learning model in an attempt to improve upon the submitted weighted f1-scores. the highest scoring model for the tamil dataset received a score equivalent to the highest scoring model presented at the sentiment analysis for dravidian languages in code-mixed text track of fire 2020. although they achieved it using a bidirectional encoder representations from transformers (bert) based model, the model presented in this paper uses word2vec and fasttext embeddings to achieve the same score. the implemented models are available on github foot_0 . ## related work analysing the sentiment of code-mixed data is important as traditional methods fail when given such data. @xcite concluded that n-grams proved to be useful in their experiments that involved multiple languages with roman script. @xcite used character n-grams in their skip-gram model. the lack of preprocessing resulted in a shorter training time and outperformed baselines that did not consider sub-word information. @xcite outperformed existing systems as well by using a sub-word based lstm architecture. their dataset consisted of 15% negative, 50% neutral and 35% positive comments. as their dataset was imbalanced like the one used in this paper, the submitted approach involved morpheme extraction as it would help in identifying the polarity of the dataset. using a hybrid architecture, @xcite an f1-score of 0.827 on the hindi-english dataset released by @xcite . after generating the sub-word level information of the input data, they used one bidirectional lstm to figure out the sentiment of the given sentence and another to select the sub-words that contributed to the overall sentiment of the sentence. in more recent work, @xcite surveyed publicly available code-mixed datasets. they noted statistics about each dataset such as vocabulary size and sentence length. @xcite used embeddings of closely related languages of the code-mixed corpus to predict named entities of the same corpus.yadav and chakraborty (2020) used multilingual embeddings to identify the sentiment of code-mixed text. as they used an unsupervised approach, they performed sentiment analysis by using the embeddings to transfer knowledge from monolingual text to their code-mixed input. @xcite designed an ensemble model consisting of four classifiers -naive bayes, support-vector machines (svm), linear regression, and stochastic gradient descent (sgd) classifiers. they tested it on a hindi-english code-mix dataset and managed to surpass the scores attained by the baseline and state-of-art systems. the proposed models test three approaches -subword analysis, word-level embeddings and machine learning algorithm based classification. the top few submissions submitted at the 'sentiment analysis for dravidian languages in code-mixed text' task proposed by fire 2020 @xcite used bert-based models. this work aims to attain the same f1-score with help of non-bertbased approaches. as dravidian code-mixed languages are under-resourced, this paper also describes the preprocessing methods used to aid in the sentiment classification of comments that contain both roman and tamil/malayalam characters. ## dataset the model has been trained, validated and tested using the tamil @xcite and @xcite datasets provided by the organizers of the dravidian code-mix fire 2020 task. the tamil code-mix dataset consists of 11,335 comments for the train set, 1,260 for the validation set and 3,149 comments for testing the model. in the malayalam code-mix dataset, there are 4,851 comments for training, 541 for validating and 1,348 for testing the model. ## result out of the models that were tested after the deadline for the fire 2020 track, the word embedding model performed the best for the tamil dataset. it attained a weighted f1-score of 0.65, which is the same as the team that scored the highest in the dravidian code-mix fire 2020 track. for the malayalam dataset, the logistic regression based model scored the highest with a weighted f1-score of 0.68. for both datasets, the linear svc model performed worse than the logistic regression model by at least 1%. it was noted that the optimal value for c was 12 for the tamil dataset and 5 for the malayalam dataset. ## conclusion this paper presents the submitted approach for the sentiment analysis for dravidian languages in code-mixed text track of forum for information retrieval evaluation (fire) 2020 and two more models to improve upon the previously attained results. the results show that the positive class in each dataset receives the highest f1-scores, regardless of the model. this is due to the higher ratio of the same as compared to the rest of the classes. comments that were not in the language of their dataset received the next highest score as their vocabulary included sub-words and words that were not a part of the rest of the datasets. although, the top ranked submissions @xcite used bert-based models for the tamil task of the dravidian languages in code-mixed text fire 2020 track, the presented work made use of wordembedding modelling and attained a weighted f1score equal to that achieved by the highest scoring team. for future work, a sarcasm detection feature could be included to avoid misclassification of comments from the positive, negative and mixedfeelings classes.