LIMSI _ UPV at S em E val-2020 Task 9: Recurrent Convolutional Neural Network for Code-mixed Sentiment Analysis (2020)

Abstract: This paper describes the participation of LIMSI_UPV team in SemEval-2020 Task 9: Sentiment Analysis for Code-Mixed Social Media Text. The proposed approach competed in SentiMix HindiEnglish subtask, that addresses the problem of predicting the sentiment of a given Hindi-English code-mixed tweet. We propose Recurrent Convolutional Neural Network that combines both the recurrent neural network and the convolutional network to better capture the semantics of the text, for code-mixed sentiment analysis. The proposed system obtained 0.69 (best run) in terms of F1 score on the given test data and achieved the 9th place (Codalab username: somban) in the SentiMix Hindi-English subtask.

## introduction in this digital era, users express their personal thoughts and opinions regarding a wide range of topics on social media platforms such as blogs, micro-blogs (e.g., twitter), and chats (e.g., whatsapp and facebook messages). multilingual societies like india with a decent amount of internet penetration widely adopted such social media platforms. however, the regional language influences the proliferation of the hindi-english code-mixed (cm) data. sentiment analysis of these end-user data from social media is a crucial resource for commerce and governance. however, in contrast to the classical sentiment analysis methods, which were originally designed for dealing with well-written product reviews, cm texts from social media often contain misspellings (often intentional), badly cased words, letter substitutions, ambiguities, non standard abbreviations, improper use of grammar, etc. cm poses several unseen difficulties to natural language processing (nlp) tasks such as word-level language identification, part-of-speech tagging, dependency parsing, machine translation and semantic processing. in the last few years, a number of workshops such as linguistic code-switching workshops 1 and shared tasks such as mixed script information retrieval @xcite have been organized due to the emerging popularity of code-mixing. to promote research in this area, task 9 of semeval-2020 was devoted to cm sentiment analysis in twitter. the goal of the task was to automatically classify the polarity of a given cm twitter post into one of the three predefined categories: positive, negative and neutral. the cm languages are english-hindi and english-spanish; for a more detailed description of the task see @xcite . in this paper, we present a deep learning approach, using a recurrent convolutional neural network for the task of automatic cm sentiment classification of tweets. the rest of the paper is structured as follows. section 2 provides background in brief. section 3 provides the system overview and section 4 describes our approach in detail. in section 5, we discuss the analysis and evaluation results for our system. we conclude our work in section 6. this work is licensed under a creative commons attribution 4.0 international licence. licence details: http:// creativecommons.org/licenses/by/4.0/ . the code of this work is available at https://github.com/somnath-banerjee/code-mixed_ sentimentanalysis. most of the earlier approaches to this problem were based on hand crafted features and sentiment lexicons @xcite . these features were then used as input to classifiers (e.g., support vector machines). however, such approaches required extensive domain knowledge, were laborious to define, and can lead to incomplete or over-specific features. recently, researchers pay their attention to sentiment polarity detection on cm data. however, a few research work have been carried out in particular hindi-english cm data with different approaches: lexicon lookup @xcite , sub-word with cnn-lstm @xcite , siamese networks @xcite , dual encoder network with features @xcite . @xcite proposed recurrent convolutional network for text classification which is a foundational task in many nlp applications. we followed this model in our task. ## system overview we are inspired by the model proposed in @xcite particularly proposed for the text classification task. the proposed model takes sequence of cm words as input and provides sentiment polarity class as output. the recurrent structure of the proposed model captures the contextual information during the learning of the word representation, and the max-pooling layer identifies the key cm words. if t , s and θ denote a cm tweet made up of sequence of cm words (@xmath0, w 2 , w 3 , . . . , w n ), the sentiment polarity class and parameters of the neural network respectively, p(s t, θ)) denotes the probability of the tweet t having sentiment polarity s, where s could be any one of sentiment polarity classes, i.e., @xmath1, negative, neutral}. in this model, a cm word is represented by combining the word and its both side contexts. if c l (w i ) and c r (w i ) denote the left-side and right-side contexts of a cm word w i , a word is represented as a concatenation of the left-side context vector c l (w i ), the word embedding e(w i ) and the right-side context vector c r (w i ). where, f : is a non-linear activation function; e(w i-1 ): the word embedding of word w i-1 ; w (l) : a matrix that transforms the hidden layer (context) into the next hidden layer; w (sl) : a matrix that is used to combine the semantic of the current word with the next word's left context. in the forward scan, the recurrent structure of the model obtains all the c l of the cm tweet, whereas, it obtains all the c r in a backward scan of the cm tweet. after obtaining the x i for the word w i , a linear transformation together with the tanh activation function is applied to x i and the result is sent to the next layer: after calculating all of the representations of words, a max-pooling layer is applied. hence, the pooling layer utilizes the output of the recurrent structure as the input. this layer attempts to find the most important latent semantic factors in the cm tweet: finally, in the output layer, a softmax function is applied that provides the sentiment polarity probabilities: ## experimental setup data: for sentiment analysis on hindi-english cm tweets, we used the dataset provided by the organizers of task 9 at semeval-2020. the training dataset consists of 14 thousand tweets. whereas, the validation dataset as well as the test dataset contain 3 thousand tweets each. the details of the dataset are given in @xcite . for this task, we did not use any external dataset. preprocessing: the cm twitter data is not formal in nature. also, often casing is not followed properly. as a result, it misleads the language processing task. therefore, we converted the entire tweets into lower case. after observing the cm dataset, we found that the urls are not contributing any information. while writing, the writer of a tweet addresses a person by mentioning the person's name followed by @ or by twitter id which starts with @. similarly, to specify a particular topic, the writer uses the topic name that starts with #. to capture the semantics and syntax of a tweet, we replaced the person and a topic being addressed with a mention and a topic token respectively. the following steps were applied on each cm tweet: embeddings: following @xcite , a lot of authors argued that word embedding plays a vital role to improve natural language task performance. hence, we experimented the use of word embeddings to improve the performance of our proposed models. using the fasttext @xcite , we prepared two embedding models: skip-gram and cbow. after empirically evaluating the performance on validation set, the embeddings' dimensionality was set to 300 for all the embeddings. the embeddings are trained on training data using the parameters: @xmath2.05, context @xmath3, @xmath4, minimal number of word @xmath5, @xmath6. experiment: we carried out two experiments with similar settings except different word embedding approaches: skip-gram for skipgrun, and cbow for cbowrun. hyper-parameters: after evaluating the model performance on the validation data, the optimal values of the hyper-parameters were set. we used the following list of hyper-parameters: learning @xmath7.6, word embedding vector @xmath8, hidden @xmath9, hidden layer @xmath10, context vector @xmath11, dropout @xmath12.1, @xmath13, loss @xmath14, and batch @xmath15. ## results and analysis the system often failed to identify the sentiment, when a long tweet consists of a number of complete or incomplete sentence with mixed sentiments. the proposed system often failed to identify the tweets that have any punctuation or delimiter to separate the sentence clauses. for example: jb koi aapke liye kam krta h aapke liye 5sal din rat ek krta h aap pe vishvas krta h aapka vishvas jit ta h jis (id:31020, gold:neutral, predicted:positive) we observed that mostly the system misclassified the positive and negative tweets as neutral and vice versa. however, we observed that there are some tweets that may arise some arguments, such as mention beautiful words and its one of my fav song (id:31662), mention sir me from bihar love you you have great job for bjp continue we are with you (id:39755), etc. tagged as neutral and our system identified as positive. identifying the polarity of an entity or entity's aspect in the tweets along with more training data could help to resolve these issues. ## conclusion this paper describes the approach we proposed for semeval-2020 task 9: sentiment analysis for cm social media text (sentimix hindi-english). in our approach, we pre-processed the cm tweets and proposed a recurrent convolutional neural network for the sentiment analysis of cm tweets. we submitted two runs and obtaining promising results: our best run obtained 0.691 of f1 averaged across the positives, negatives and the neutral. we observed that the proposed architecture occasionally strives to separate the positive and negative polarities from the neutral and vice versa. for future work, we will explore the performance of our model with larger corpora against the testing set. also, we would like to investigate other embedding choices such as bert @xcite . moreover, due to the impact that irony and sarcasm have on sentiment analysis @xcite it would be interesting to apply deep learning techniques to detect irony @xcite but in a code-mixed scenario.