IUST at S em E val-2020 Task 9: Sentiment Analysis for Code-Mixed Social Media Text Using Deep Neural Networks and Linear Baselines (2020)

Abstract: Sentiment Analysis is a well-studied field of Natural Language Processing. However, the rapid growth of social media and noisy content within them poses significant challenges in addressing this problem with well-established methods and tools. One of these challenges is code-mixing, which means using different languages to convey thoughts in social media texts. Our group, with the name of IUST(username: TAHA), participated at the SemEval-2020 shared task 9 on Sentiment Analysis for Code-Mixed Social Media Text, and we have attempted to develop a system to predict the sentiment of a given code-mixed tweet. We used different preprocessing techniques and proposed to use different methods that vary from NBSVM to more complicated deep neural network models. Our best performing method obtains an F1 score of 0.751 for the Spanish-English sub-task and 0.706 over the Hindi-English sub-task.

## introduction sentiment analysis identification is a sub-field of natural language processing that explores the automatic inference for fine-grained opinion polarity of textual data. the recent growth of social media, review forums and text messaging platforms create a surge in the amount of user-generated textual data, and so increased the urgent need for automatic opinion extraction. however this evolution created many opportunities for language technology and researchers, it provided verity of new challenges, namely, spelling errors, creative invented spelling ("iz" for "is"), abbreviation ("lol" for "laugh out loud"), meta tags(hashtags) and code-mixing @xcite . non-english speakers frequently use multiple languages to express their feelings, which they know as code-mixing. speakers or writers tend to shift from one language to another either to express their feelings adequately, to show like-mindedness with a group, to distinguish oneself, to discuss a specific topic, or to look impressive to their audience. the semeval-2020 shared task 9 @xcite , which is part of the semeval-2020 workshop, focused on coping with this challenge. this task focus is automatic sentiment analysis in a code-mixed social media text. the task consists of two subtasks, spanish-english(spanglish) and hindi-english(hinglish) code-mixing, with over 30 groups who participate in each sub-task. this article presents a system that we have implemented for predicting the sentiment of a given codemixed tweet. the system was developed for both subtasks. ## related work sentiment analysis is a well-established and well-studied topic in social media analysis and natural language processing(nlp); however, most of the effort has been applied to addressing this problem in the monolingual context. since english is an international language and is the second most spoken language globally, it is most likely to be code-mixed with other languages. hence, most of the works in the code-mixed criteria are focused on the combination of english with other languages. @xcite had divided the code-mixed sentiment analysis of english and indian languages problem, into the language identification and sentiment analysis sub-tasks. first, they used wordlevel language labeling and then applied sentiwordnet to analyze the semantic of each word. overal sentiments of the sentence had obtained from all the words' sentiments. ghosh (2017) adopted a multilayer perceptron with varieties of extracted features such as part-of-speech tags, frequency of code switches, the density of curse words, and the number of slang words. parvalika (2017) proposed a method based on a lexicon-based and machine learning approaches to address hindi-english sentiment analysis. in the lexicon-based method, they used dictionaries of words annotated with their polarity to classify text. in the machine learning-based part, first, they did data cleaning and replace slang words with their associated meaning. then, naive bayes(nb), support vector machine(svm), decision tree(dt), random tree(rt), multilayer perceptron were applied for text classification purposes. @xcite addressed the same problem with sub-word level representations in lstm architecture. they applied character-level embedding followed by convolution with a filter length of 3 in order to obtain sub-word level representations. lstm was utilized to model the relationships between obtained features. the lstm output feeds to a fully connected layer to calculate the final label. lal (2019) proposed a model that consists of three different components. the first component prepared the sub-word level representations. then the output was feed to the second component, which comprises two different bilstm. one bilstm aims to learn a representation for the overall sentiment of the sentence, and the other one, which is followed by an affixed attention mechanism, enables the selection of subwords that contribute the most towards the input text's sentiment. finally, the concatenation of two bilstm aligns with linguistic features feed to a fully connected layer to predict sentence polarity. in order to address the bambara-french code-mixed sentiment analysis problem in facebook comments, @xcite employed a variety of deep learning methods, including lstm, bilstm, cnn, and their combinations. character and word embedding produced from fixed indexes were used due to the lake of pre-trained word vectors for the bambara language. vilares (2015) investigated sentiment analysis for english and spanish language separately and their combination on twitter. they extracted a set of linguistic features. they also make use of dependency parsers to achieve syntactic features. they developed an utterly monolingual model, a pipeline that adopted language identification techniques to discover the language of texts, and a multilingual model trained on spanish-english texts. for the word embedding phase in resolving the spanish-english code-mixed sentiment analysis problem, yadav (2020) used various methods. in the first approach, they separately learn word embedding for each language and then used supervised and unsupervised approaches to learn a transformation matrix to map representation from one language to the representation of the other language. in the second method, they trained word, and subword representation on corpus contains text from both languages and their combination. last, they adopted a machine translation approach to learning the cross-lingual sentence-level representation. they achieved the best result with the combination of subword representation and lstm. ## data two corpora were used in code-mixed sentiment analysis shared task. the spanish-english dataset consists of 15000 data samples for the train and 3789 for the test. on the other hand, the hindi-english dataset contains total 15131 data samples for the train and 3000 for the test set. the sentiment labels are positive, negative, or neutral. besides the sentiment labels, the language labels at the word level have been provided. the distribution of both language labels for each dataset is shown in figure 1 shows the overlap between the unique tokens between the first and second languages. there are 614 words in the spanish-english dataset, which have the same spelling in both languages. this number is much higher for the hindi-english dataset. it contains 5884 words with similar spelling for both hindi and english languages, and approximately, there is an 11% overlap between english and hindi vocabulary. figure 1 : vocabulary overlap for language pairs for spanish-english and hindi-english datasets. ## methodology in this section, we describe the techniques and models that we applied to address the code-mixed sentiment analysis problem. ## results for all models, we used inputs with and without preprocessing. also, we experiment with different parameter and reports the set of parameters with the best performance. moreover, we tested all the models, as mentioned earlier, for both hindi-english and spanish-english datasets. the performance of nbsvm was quite impressive. we used the character n-gram with a range of 2 to 6 and the tfidt matrix. the language independence characteristic of this model makes it a strong candidate for the code-mixed classification tasks. our experiment with the distilbert model was not satisfying. however, multilingual bert performed slightly better. ## conclusion our proposed methods ranked 6 out of 62 groups for the hindi-english dataset and ranked 7 out of 29 for the spanish-english dataset. this result shows the strength of the combination of nbsvm and tf-idf as a language independence model. also, our experiment shows the substantial adverse effect of ignoring one of the language's representation. for future work, we are going to utilize both languages pre-trained embedding. averaging is one way to combine both embeddings. however, the different output space of both embedding could be challenging. also, the attention mechanism could be used to give the model the ability to decide the importance of each language representation within each sentence.