C1 at S em E val-2020 Task 9: S enti M ix: Sentiment Analysis for Code-Mixed Social Media Text Using Feature Engineering (2020)

Abstract: In today’s interconnected and multilingual world, code-mixing of languages on social media is a common occurrence. While many Natural Language Processing (NLP) tasks like sentiment analysis are mature and well designed for monolingual text, techniques to apply these tasks to code-mixed text still warrant exploration. This paper describes our feature engineering approach to sentiment analysis in code-mixed social media text for SemEval-2020 Task 9: SentiMix. We tackle this problem by leveraging a set of hand-engineered lexical, sentiment, and metadata fea- tures to design a classifier that can disambiguate between “positive”, “negative” and “neutral” sentiment. With this model we are able to obtain a weighted F1 score of 0.65 for the “Hinglish” task and 0.63 for the “Spanglish” tasks.

## introduction social media has grown exponentially in the last decade and has given rise to multilingual online communication and the consumption of media. code mixing is the phenomenon of embedding linguistic units such as phrases, words, or morphemes of one language into an utterance of another. code mixing is prevalent in online discourse, where bilingual speakers mix english with their native language. for instance, bilingual speakers of hindi and spanish languages, which are the 3rd and 4th most spoken languages in the world respectively, frequently mix english with hindi and spanish in spoken language as well as online social media. the growing presence of "hinglish" and "spanglish" in social media is evidence that this is a growing area of research as traditional nlp tasks like sentiment analysis heavily rely on monolingual resources. while there are many cutting edge techniques to apply common nlp tasks to monolingual text, the task of sentiment analysis, in particular, has not been explored for multilingual code-mixed texts. conventional nlp tools make use of monolingual resources to operate on code-mixed text, which limits them to properly handle issues like word-level code-mixing. additionally, code-mixed texts typically exist on social media which has a conjunction of other features like incorrect spelling, use of slang, and abbreviations to name a few. while code-mixing is an active area of research, correctly annotated data is still scarce. it is particularly difficult to mine a small subset of tweets that may or may not be code-mixed and then apply a sentiment score to them. the organizers have solved this by releasing around 20,000 tweets @xcite with unique sentiment scores and word-level tagging of the language. for evaluation and ranking, the organizers used a weighted f1 score across "positive", "negative" and "neutral" classes. while deep learning techniques are state of the art for this task, the contribution of this paper is to demonstrate how content-rich features, when applied to a classifier, can give us strong and comparable results. some of the features we explore are lexical features like n-grams, metadata features like word frequency, and sentiment-based features like profanity count. ## related work code-mixed sentiment analysis is an active field of research. for monolingual sentiment analysis, recurrent neural networks (rnn) and other more complex deep learning models have been successful. @xcite gave us a significant breakthrough by using compositional vector representations. with regards to hindi-english sentiment analysis, the shared task @xcite reported accuracies of different systems submitted on sentiment analysis for tweets in three major indian languages: hindi, bengali, and tamil. recently @xcite released a dataset sourced from facebook and proposed a deep learning approach to hindi-english sentiment extraction. a variety of traditional learning algorithms like naive bayes, support vector machines and decision trees were applied to this dataset. the winning algorithm from the sail-2015 shared task was a naive bayes based system. more recent work by @xcite involves a hierarchical multitask neural architecture with the lower layers performing syntactic tasks, and the higher layers performing the more involved semantic tasks while using the lower layer predictions. for the hinglish dataset, the organizers have released a total of 20,000 manually labeled tweets with 14,000 being in the training dataset, 3,000 being part of the validation dataset and 3,000 being part of a holdout test dataset. additionally, for the spanglish dataset, they released 12,002 tweets for the training dataset, 2,998 tweets for the validation dataset and 3,788 tweets for the test dataset. to ensure the accuracy of the dataset labels, the organizers employ a semi-automatic annotation methodology. they initially use word-level identifiers and then baseline sentiment analysis to obtain the initial labels. subsequently, manual evaluation is done by at least two annotators, tweets with low confidence are discarded. while this seems to be a robust system in some cases the word tags are incorrect with words like 'maybe' being tagged as a hindi word, when we can safely assume that it is from the english language. additionally, the dataset contains urls, hashtags, usernames and emoticons. the average length for tweets in the "hinglish" dataset is 26 tokens and 15 tokens for the "spanglish" dataset. the data was provided in the conll format with each tweet having a sentiment of "positive", "negative" or "neutral". figure 1 gives us an overview of the class distribution of these datasets. as we can see the "spanglish" dataset is unbalanced with the majority of the samples being in the "positive" and "neutral" datasets. from the examples in ## methodology we extracted different hand engineered features like word n-grams, sentiment polarity, punctuation frequency, word frequency, emoji frequency and profanity level from tweets. we weighed n-gram features using their term frequency -inverse document frequency (tf-idf) scores. we then built a logistic regression classification model using scikit-learn @xcite library. ## results and analysis