Gundapusunil at S em E val-2020 Task 9: Syntactic Semantic LSTM Architecture for SENTI ment Analysis of Code- MIX ed Data (2020)

Abstract: The phenomenon of mixing the vocabulary and syntax of multiple languages within the same utterance is called Code-Mixing. This is more evident in multilingual societies. In this paper, we have developed a system for SemEval 2020: Task 9 on Sentiment Analysis of Hindi-English code-mixed social media text. Our system first generates two types of embeddings for the social media text. In those, the first one is character level embeddings to encode the character level information and to handle the out-of-vocabulary entries and the second one is FastText word embeddings for capturing morphology and semantics. These two embeddings were passed to the LSTM network and the system outperformed the baseline model.

## introduction code-mixing is a phenomenon which is evident in multilingual societies @xcite . it reflects the use of distinct grammatical systems and vocabulary of the languages being used simultaneously in a single utterance or conversation. this technique used in communication commonly is widely found today in popular social media platforms like twitter, facebook, instagram in the form of posts, comments, replies, especially in chats. this is evident in multilingual societies like india, canada, ireland, south africa, switzerland, and many others. india has officially recognized 22 regional languages foot_0 . so, in multilingual societies like india, most of the social media users predominantly integrate the well-known language, like english, with their native languages. 560 million internet users foot_1 in india exchange information by mixing their regional languages with prominent language like english, which produces a huge amount of code-mixed social media corpus. one such trending combination is the mixing of hindi and english with the output in hinglish foot_2 (hi-en) code-mixed data. consider the example sentence which illustrates the code-mixing phenomenon being addressed in this paper. "congratulations/eng sir/eng ji/hin dobara/hin pm/eng banee/hin ki/hin hardik/hin subhkamnaye/hin aapko/hin ./o". (translation into english: congratulations sir, best wishes to become prime minister again). the words followed by language tags /hin, /eng, and /o correspond to hindi, english and other respectively. sentiment analysis of code-mixed data has become a prominent research area in recent times in the field of nlp. but identifying the sentiment in code-mixed data is hard since it poses the following challenges: (i) romanized code-mixed data is noisy and ambiguous in nature. (ii) accessible datasets are smaller in size to tune neural networks. (iii) frequent occurence of non-standard spellings (such as pyaaarr, goooood). (iv) the phrase/word contractions (cmng for coming, idk for i don't know). (v) spelling variations. a single word pyaar (love), can be written as "piyar", "pyaarrrr", "peyar", "pyar", or "piyaar", etc. to handle these challenges we postulate that fasttext embeddings enrich the word vectors with sub-word information and that character level embeddings should be able to assist deep learning models to handle unknown hindi words. in this paper, we propose models to predict the sentiment label of a given code-mixed tweet/text. the sentiment labels are positive, negative, or neutral and the code-mixed languages selected are hindi and english. this task is conducted in codalab foot_3 website and our codalab username is gundapusunil. all our models are trained using only the trained dataset provided by sentimix organizers. we started experiments with traditional machine learning models like logistic regression, support vector machines but the f1-score on development set was below 0.63 then we moved to complex models like long short term memory (lstm) based models with different types of word embeddings where we were able to conquer the baseline model. our paper is divided into the following sections: we begin with an introduction to code-mixing and its challenges in section 1. related work of various code-mixing strategies demonstrated in section 2. section 3 present the description of sentimix dataset. we then discuss the pre-processing steps and compare machine learning and deep learning approaches with baseline model results in section 4. the results are reported in section 5, and section 6 concludes the paper. ## related work several studies have been made in the areas of sentiment analysis and code-mixed data. one of the earlier studies on code-mixed data was proposed by @xcite with the goal of language identification in which it was stated that the structure of the language is procured by learning the language structure from the given text and informant. @xcite described the structure of multilingual languages and language dependency in linguistic convergence of code-mixing from an indian perspective. sentiwordnet for english language introduced by @xcite , became the primary source for all sense based lexical analysis and opinion classification. later, researchers have extended the work on machine learning and sentiment analysis methods for indian languages. in their study @xcite used the svm and naıve bayes classifiers to label millions of comments for sentiment polarity. r. @xcite developed a lexicon-based sentiment analyzer for product reviews and the same subjective lexicon-based model has been extended to punjabi language. for malayalam movie reviews d. s. nair, j. p. jayan et al. @xcite initially came up with a rule based system for sentiment analysis, later improved the system with the support of the machine learning algorithms. mike 2015 sentiment analysis task for hindi and bengali used the multinomial naive bayes classifier with the features for building the vector space constrained by filtering the words based on wordnet. in recent years, researchers have seen a huge improvement in the task of sentiment analysis of english as well as hinglish using deep neural networks. m.g. jhanwar, a. das (2018) proposed an ensemble of character-trigrams based lstm model and a n-grams based multinomial naive bayes model to classify the sentiments of hinglish code-mixed data. @xcite addressed the performance of distributed representation methods in sentiment analysis and reported comparisons among different machine learning and deep learning techniques. other attempts include using sub-word level compositions with lstms to capture sentiment at the morpheme level @xcite . we attempted to perform semeval 2020 task-9 @xcite with various classification and deep learning models to analyze the results and also how such models contributed to a great advance in this task. ## dataset in this paper, we used the dataset provided by task 3: sentimix in semeval 2020. the corpus contains a total of 20000 tweets and it is sub-divided into three sets (train, validation and test). each corpus except test contains code-mixed tweets along with their corresponding sentiment labels. these code-mixed tweets are tokenized into tokens. and the tokens of each tweet are separated by a new line. each token is manually annotated with a language identification tag which are: hindi (hin), english (eng), other (o). ## system architecture in this section, we present our models that are trained and validated on the sentimix dataset described in the previous section. we compare our approach with machine learning and neural network based baselines. the full code of system architecture can be found at github foot_4 . ## conclusion in this paper, we experimented the code-mixed dataset with various machine learning and deep learning models. we see that the lstm models performed far better than traditional ml methods. in the first phase of the sentimix competition (development set), we were able to achieve a score of 0.6357. but in the second phase (test dataset), our best score was only 0.6758. after competition we attain the f1-score of 0.6789 by changing few parameters like learning rate and number of lstm's. till now we handled problems like unseen words, spelling variations, dataset imbalance, emojis, short form of words, etc.