Zyy1510 Team at S em E val-2020 Task 9: Sentiment Analysis for Code-Mixed Social Media Text with Sub-word Level Representations (2020)

Abstract: This paper reports the zyy1510 team’s work in the International Workshop on Semantic Evaluation (SemEval-2020) shared task on Sentiment analysis for Code-Mixed (Hindi-English, English-Spanish) Social Media Text. The purpose of this task is to determine the polarity of the text, dividing it into one of the three labels positive, negative and neutral. To achieve this goal, we propose an ensemble model of word n-grams-based Multinomial Naive Bayes (MNB) and sub-word level representations in LSTM (Sub-word LSTM) to identify the sentiments of code-mixed data of Hindi-English and English-Spanish. This ensemble model combines the advantage of rich sequential patterns and the intermediate features after convolution from the LSTM model, and the polarity of keywords from the MNB model to obtain the final sentiment score. We have tested our system on Hindi-English and English-Spanish code-mixed social media data sets released for the task. Our model achieves the F1 score of 0.647 in the Hindi-English task and 0.682 in the English-Spanish task, respectively.

## introduction mixing language, also known as code-mixing, is a norm in multilingual societies. many multilingual people tend to be code-mixed by using english-based speech types and the insertion of english into their main language @xcite , which share their views on social media by combining local and english languages, creating lots of code-mixed text such as hindi-english and english-spanish @xcite . today, many organizations rely heavily on sentiment analysis of social media texts for product performance and consider user feedback when upgrading to newer versions @xcite . the government can predict people's emotions and know people's opinions on the new policy and so on. code-mixing @xcite is a relatively new field compared to the general field of sentiment analysis @xcite . social media code-mixed texts generally have three forms: i) mixed script: a combination of the native-roman script; ii) code-mixed script: a script written in roman script in native and english languages; iii) native script: local languages written in native languages. this type of text needs to be handled differently, which is very different from traditional english texts @xcite . beyond some of the challenges of general sentiment analysis, code-mixed texts have some unseen difficulties in natural language processing (nlp) tasks. traditional nlp systems heavily rely on monolingual resources to address code-mixed text, which limits their ability to handle problems such as english-based speech input, word-level code-mixing, et al @xcite . furthermore, there are several variations when switching from a phonetic language into a roman script (jhanwar and das, 2018). to solve this problem, we preprocess the text and normalize irregular words. before we preprocess the text, we also need to eliminate the noise in the text, and translate the abbreviations into the appropriate regular words, and perform a clustering algorithm to get the most suitable one of the last few variants when transliterating non-roman script code-mixed data as roman scripts in preprocessing step. in our work, we introduce a sentiment analysis (sa) system, an ensemble model of word n-gramsbased multinomial naive bayes(mnb) and sub-word level representations in lstm (sub-word lstm) @xcite . for indian social media text which is developed for the semeval-2020 shared task on sentiment analysis for code-mixed (hindi-english, english-spanish) social media text aims to detect the sentiment polarity @xcite of the code-mixed text written in two different languages, hindi and spanish mixed with english. the traditional method mnb captures low-level word-groups of keywords to make up for grammatical inconsistencies, while the sub-word lstm model encodes the rich sequential patterns in sparse and unstable text @xcite . our model achieves the f1 scores of 0.647 in the hindi-english task and 0.682 in the english-spanish task, respectively. the implementation of our system is made available via github 2 . ## related work recently, research on emotion and mood analysis in texts became increasingly common, in part because of the availability of new sources of subjective information on the web. @xcite is one of the earliest in the area of sentiment classification. it is concerned with the actual classification and segregation of terms with emotional connotations. @xcite described language identification in the first shared task of the code-switched data held at emnlp 2014. @xcite analyzed the data on facebook posts generated by english-hindi bilingual users. sail 2015 contest co-organized with mike 2015 considered sentiment analysis of tweets in indian languages @xcite . @xcite introduced learning sub-word level representations in lstm architecture. @xcite tried to use machine learning methods to automatically extract sentiment (positive or negative) from facebook posts. @xcite addressed the performance of distributed representation methods for bengali-english and hindi-english languages in sentiment analysis tasks. @xcite used a machine learning algorithm called multinomial naive bayes trained by using n-gram and sentiwordnet features, they also used a small sentiwordnet for english and bengali without using any sentiwordnet for hindi language, hindi-english and bengali-english code-mixed data. an ensemble model of character tri-grams based lstm model and word n-grams based multinomial naive bayes (mnb) model to classify the sentiments of hindi-english code-mixed data was introduced by (jhanwar and das, 2018). (ansari and govilkar, 2018) designed the system which classifies hindi as well as marathi text transliterated (romanized) documents automatically using supervised learning methods (knn), naive bayes and support vector machine (svm) and ontology-based classification. @xcite presented a hybrid architecture for the task of sentiment analysis of english-hindi code-mixed data. @xcite prepared gold standard bengali-english code-mixed data with language and polarity tag for sentiment analysis purposes, and a hybrid system combining rule-based and supervised models were developed for both languages. ## dataset a recent shared task was conducted by international workshop on semantic evaluation 2020 on nlp (semeval-2020), for sentiment analysis of transliterated social media text. the organizer of semeval-2020 provided the code-mixed data of hindi-english and spanish-english. the training and validation tweets were labeled one of the three labels -positive, negative and neutral. there are many inherent challenges of the code-mixed data as described previously. examples like abbreviations of words ('please' to 'plz') and non-standard spellings (such as 'suppeerrrr' or 'timeeeeeee'). and there are several variations when switching from a phonetic language into a roman script, as illustrated in before training, we preprocessed the code-mixed raw data. we replaced the link in the data as the url and removed the punctuation, stop-words and the useless emoji. we tried to make the data noiseless. we also found that some certain characters appear multiple times in a word. for instance, lol (meaning laughing out loud) can be written as loool, looool or looooool. we used a clustering algorithm to process it as lool in the pre-processing stage. and we divided the hash form into the appropriate form (happybirthdaysonakshisinha as happy birthday sonakshi sinha). in addition, we transliterated non-roman script ( , means wishes) code-mixed data as roman scripts (mubarak). here we chose a clustering algorithm(k-means clustering algorithm) to get the most suitable variant @xcite . all text is converted to lower-case and then fed to the classifier. ## system description our proposed system architecture is shown in figure 1 , which is an ensemble model of word n-gramsbased multinomial naive bayes (mnb) and sub-word level representations in lstm (sub-word lstm) to identify the sentiments of hindi-english and english-spanish code-mixed data into one of the sentiment classes positive, negative or neutral. after pre-processing the sentence, we generate word-based uni-gram and bi-gram features of the sentence and then fed them to the mnb classifier. finally, it outputs the probability of the sentence belonging to each class. in our deep learning model, we feed an embedded matrix with length of 128 to the lstm cell. we use middle level representations of the sub-word that the filter learned in the convolution operation. it propagates serviceable information with lstm and obtains the final score of the text as illustrated in then the information is fed into a full connection (fc) layer, which achieves the interactions between these features and classes @xcite . the soft maximum activation function was used to output the correct probability value. we use adamax @xcite as the optimizer, a variant of adam, to train this setup. the optimal hyperparameter configuration of the model is shown in ## experiments detail this section presents the results and compares them to several baselines. we submitted a run for each language: (1) one for hindi-english and (2) one for spanish-english. the final ranking for all participating systems would be based on the f1 score averaged across the positives, negatives, and the neutral. we experimented an ensemble model of word-n-grams based multinomial naive bayes (mnb) and sub-word level representations in lstm (sub-word lstm) to identify the sentiments of code-mixed data of hindi-english and english-spanish. we chose the model with the highest combined probability by multiplying the output probability of the two models of each class. we observed that mnb (unigram+bigram) performed better for the sparse and inconsistent code-mixed data, especially rare keywords like 'fadu', meaning awesome in english (jhanwar and das, 2018) than svm (unigram+bigram). because the n-gram-based mnb model can successfully capture the unusual keywords. the sub-word lstm can extract better sequence information for long sentences. that's why we decided to use the ensemble model. ## conclusion and future work social media is becoming increasingly influential in people's lives. people in different positions and occupations express their views and attitudes on a certain topic. some researchers are fascinated for the sentiment analysis of social media text. in this paper, we proposed an ensemble model for sentiment analysis of code-mixed data for hindi-english and english-spanish. in the future, we're going to put emotional information into the system and also introduce new networks such as transformers, bert, attention mechanism etc.