WESSA at S em E val-2020 Task 9: Code-Mixed Sentiment Analysis Using Transformers (2020)

Abstract: In this paper, we describe our system submitted for SemEval 2020 Task 9, Sentiment Analysis for Code-Mixed Social Media Text alongside other experiments. Our best performing system is a Transfer Learning-based model that fine-tunes XLM-RoBERTa, a transformer-based multilingual masked language model, on monolingual English and Spanish data and Spanish-English code-mixed data. Our system outperforms the official task baseline by achieving a 70.1% average F1-Score on the official leaderboard using the test set. For later submissions, our system manages to achieve a 75.9% average F1-Score on the test set using CodaLab username “ahmed0sultan”.

## introduction microblogging websites have been a huge source of data containing different kinds of information. since the users on these microblogging websites tend to write informal real-time messages, they also tend to mix languages as they are just being spontaneous and want to ease the communication or they are just multilingual or non-native language speakers who mix between their native language and the language they are trying to use @xcite . this type of writing is called code-mixing or code-switching and it could be defined as the phenomenon of mixing the vocabulary and syntax of multiple languages in the same sentence @xcite . sentiment analysis (sa) is the task of detecting, extracting, and classifying sentiment and opinions @xcite . sa can help in structuring data of public opinions about products, brands or any topic that people can express opinions about, to be used in a very wide set of practical applications varying from political use, e.g. monitoring public events @xcite to commercial use, e.g. making decisions in the stock market @xcite . the task of monolingual sentiment analysis has been a well-studied topic in the literature over the past few decades. however, little attention has been directed to sa based on code-mixed data. in semeval-2020 task 9: sentiment analysis for code-mixed social media text @xcite , the organizers provide a dataset of code-mixed tweets with word-level language labels that we will explore in section 2.1, and with the following sentiment labels: positive, negative, neutral. given a code-mixed text, the task is to classify the overall sentiment of the input text to one of the three sentiment labels mentioned above. the official evaluation metric for this task is average f1-score. we report the experiments made only on spanish-english (spanglish) data, whereas semeval-2020 task 9 contains hindi-english (hinglish) data as well. the challenges of this shared task could be summarized as follows: a) the relatively small dataset provided makes it hard to train complex models b) the target classes distribution is imbalanced in the training data c) the characteristics of social media text pose difficulties such as out-of-vocabulary words and ungrammatical sentences due to the spontaneous and informal writing with lots of extra embedded information in a single sentence (e.g. hashtags, emojis, or repeated characters in a word) some of this embedded information can be utilized to make a better predictive model and others may hurt the model prediction badly. we conducted several experiments to tackle this problem. we used a linear svm, logistic regression, and multinomial naive bayes models with tf-idf feature vectors as an input. we also used xlm-roberta @xcite , a transformer-based multilingual masked language model which is trained on 100 languages, fine-tuned on our downstream sa task which achieves our highest score outperforming the official baseline. the rest of the paper is structured as follows: section 2 introduces some background about the task, an overview of the dataset, and related work. we describe the applied preprocessing steps and the experiments in section 3. we report the results in section 4. and section 5 summarizes our work. ## background whereas the task of sentiment analysis has been extensively explored in the past few decades, there has been less work on code-mixed sentiment analysis (cmsa). given a sentence in the code-mixed language, the task is to predict its sentiment polarity (i.e. positive, negative, or neutral). for example, the spanglish tweet "ha u know is true parece ke te esta dando un atake d" should be classified as a positive sentiment. ## system description in this section, we are proposing different approaches to tackle the problem of identifying the sentiment in code-mixed text. ## conclusion in this paper, we described our experimentations on semeval 2020 task 9, sentiment analysis for code-mixed social media text. we presented four main systems; logistic regression, multinomial naive bayes, svm with linear kernel, and a fne-tuned xlm-roberta. the submitted xlm-roberta system outperformed the baseline on the test set with an average f1-score of 70.1% and 52.20% on the development set. whereas the enhanced approach of the xlmroberta system outperformed the submitted version by achieving a 54.7% average f1-score on the development set. future work includes working on using multilingual word embeddings that's aligned in the same space like @xcite .