ULD @ NUIG at S em E val-2020 Task 9: Generative Morphemes with an Attention Model for Sentiment Analysis in Code-Mixed Text (2020)

Abstract: Code mixing is a common phenomena in multilingual societies where people switch from one language to another for various reasons. Recent advances in public communication over different social media sites have led to an increase in the frequency of code-mixed usage in written language. In this paper, we present the Generative Morphemes with Attention (GenMA) Model sentiment analysis system contributed to SemEval 2020 Task 9 SentiMix. The system aims to predict the sentiments of the given English-Hindi code-mixed tweets without using word-level language tags instead inferring this automatically using a morphological model. The system is based on a novel deep neural network (DNN) architecture, which has outperformed the baseline F1-score on the test data-set as well as the validation data-set. Our results can be found under the user name “koustava” on the “Sentimix Hindi English” page.

## introduction sentiment analysis refers to a process of predicting the emotion content from a given text. sentiment analysis is usually seen as a categorization problem over a variable with three values: positive, negative, neutral @xcite . with the increase in the popularity of social media such as twitter, a new area of study to the field of natural language processing and thus, sentiment analysis has been explored. most of the data extracted from social media are code-mixed @xcite , which have become a common approach in most cases but also pose unique challenges. analysis of short texts from micro-blogging platforms such as twitter is in high demand as the analysis of these text distill and evaluate the moods and the sentiment of the users and are very useful for different organisations, be it government or business or ngo. sentiment analysis for indian code-mixed languages is relatively new @xcite @xcite . the significant difference in style of language, orthography @xcite and grammar used in tweets presents specific challenges for english-hindi code-mixed data. in this paper we aim to introduce a novel deep neural network system which was submitted for semeval 2020 task 9, sub task a for english-hindi data @xcite . we will also compare the system with other state-of-the-art systems and describe how the system has outperformed others. the systems were trained using only the twitter data provided by the organisers excluding the word-level language tags provided in the data. ## related work although the field of sentiment analysis is growing and several systems have advanced the state-of-the-art, the overall performance of systems to predict sentiment in code-mixed data is low. @xcite predicted overall sentiment score for hindi-english code-mixed data using a lexicon based approach. @xcite were the first to look at the task as a query-driven classification problem. a hindi-english data-set was introduce by @xcite for sentiment analysis and they performed empirical analysis comparing the performance of various state of the art models in sentiment analysis. they also introduced a sub-word level representation in an lstm model instead of character or word level representation. dos santos and gatti (2014) proposed a deep convolutional neural network that exploits character level and sentence level information to predict the sentiments in short texts. all these previous experiments were dependent on the word-level language tags, and this is a disadvantage as it is time-consuming to annotate at the word level. in our approach, we create a model without the need for word-level annotation. ## dataset the dataset used for the current task is provided by sentimix english-hindi task 9 in semeval-2020 @xcite . it consists of english-hindi code-mixed tweets annotated with sentiment labels: positive, negative, or neutral. besides the sentiment labels the data-set also includes word-level language tags, which are en (english), hi (hindi), mixed, and univ (symbols, @ mentions, hashtags). as it is very common for twitter data to have other forms of text such as urls and emoticons, this data-set too contains emojis such as and urls. the pre-processing removes the word-level language tags. we normalize the data for training the support vector machine (svm) and deep neural network (dnn), by lower-casing all the tweets and removing punctuation, emojis and urls. after converting all the tweets into lower case, extra spaces were removed from the tweets. the tweets are tokenized into characters, where each character has been mapped to an index number. the character-index mapping is created with the help of the keras tokenizer package foot_1 . ## results overall we see varying performance across the classifier, with some performing much better out-of-sample than others. the state-of-the-art character cnn model has performed better than the svm model. one of the main reasons for a cnn to perform better than svm is that a cnn is capable of identifying the features of the sentence with the help of neural model weight distribution. it also takes special characters into account which make the sentence embedding more robust to work on. on the other hand, the hyper-tuning settings of the tf-idf vectors could be the cause of lower performance of the svm. our genma model has outperformed all classical models as well as the state-of-the-art character cnn model as it considers a sentence composed of a different set of morphemes. the individual results on three different sentiment classes show that the model outperforms the other two models while recognizing individual classes whereas the svm model recognizes neutral classes better than the cnn model. our model has achieved 0.68 f1-score in the test set which is 7 percent better than the svm and 5 percent better than the character cnn model. ## discussion our proposed genma model outperforms other models as it is capable of generating new morphemes out of neighbor characters and it identifies the essential morphemes to classify a sentence. the two main advantages of our model are: in the generated artificial morpheme, some morphemes put more emphasis on sentence polarity (an example is morpheme "ote", which weights 5 times (0.061) than the normal morpheme "arn" (0.011)). the softmax attention weights are able to rank character importance from high to low. ## conclusion in this paper we have proposed a novel deep neural model which has outperformed the baseline scores on code-mixed data proposed in @xcite and state-of-the-art models discussed in section 5. our model is capable of classifying the sentiment of the sentences without considering language difference between words in the sentences with an f1-score of 0.68 on the test data. future work may reveal how to capture sentiment based on emojis that are widely used in tweets. one of our settings is artificial morpheme generation for the hindi and english dataset. but we have not explored this method in the context of morphologically complex code-mixed datasets. we will aim to implement the model in the complex code-mixed dataset in the future. we will also try to capture word level information of code-mixed sentences without language identity to understand what the important key words are to classify sentences.