UPB at S em E val-2020 Task 9: Identifying Sentiment in Code-Mixed Social Media Texts Using Transformers and Multi-Task Learning (2020)

Abstract: Sentiment analysis is a process widely used in opinion mining campaigns conducted today. This phenomenon presents applications in a variety of fields, especially in collecting information related to the attitude or satisfaction of users concerning a particular subject. However, the task of managing such a process becomes noticeably more difficult when it is applied in cultures that tend to combine two languages in order to express ideas and thoughts. By interleaving words from two languages, the user can express with ease, but at the cost of making the text far less intelligible for those who are not familiar with this technique, but also for standard opinion mining algorithms. In this paper, we describe the systems developed by our team for SemEval-2020 Task 9 that aims to cover two well-known code-mixed languages: Hindi-English and Spanish-English. We intend to solve this issue by introducing a solution that takes advantage of several neural network approaches, as well as pre-trained word embeddings. Our approach (multlingual BERT) achieves promising performance on the Hindi-English task, with an average F1-score of 0.6850, registered on the competition leaderboard, ranking our team 16 out of 62 participants. For the Spanish-English task, we obtained an average F1-score of 0.7064 ranking our team 17th out of 29 participants by using another multilingual Transformer-based model, XLM-RoBERTa.

## introduction a series of countries from our world are multilingual, which implies that there are multiple languages spoken by their population. people tend to mix them at the phrase or sentence level in order to express ideas with ease, thus creating a phenomenon called code-mixing or code-switching. as it is expected, this embedding of a language into another one makes its appearance in the virtual space, as well. for example, twitter users combine hindi or spanish phrases with english words, thus creating a bilingual phrase that can lead to understanding problems for non-natives. however, the virtual space adds more layers of difficulty in identifying the sentiment of the author. usually, social media users tend to adopt phonetic typing, which implies that the words will not take their original form, they will be adapted such that it will be faster to express the main idea. as a particular case, for the hindi-english users, a new problem arises: hindi and english use different alphabets, which determines the user to romanize the hindi words such that both languages will use the same alphabet throughout the text. at the same time, social media users tend to express their sentiments by repeating certain vocals in words. furthermore, they use emojis, which will add an extra layer of complexity for analyzing the text. this entire process creates new opportunities for research, given the importance of sentiment analysis in this area. the semeval-2020 task 9: sentiment analysis for code-mixed social media text @xcite challenges the research community to solve the previously mentioned problem by introducing two subtasks, focusing on three of the world's most spoken languages 1 : hindi and spanish, alongside english. we proposed a series of neural models that intend to solve this issue, contributing under the this work is licensed under a creative commons attribution 4.0 international licence. licence details: http://creativecommons.org/licenses/by/4.0/ . 1 https://www.visualcapitalist.com/100-most-spoken-languages/ usernames eduardgzaharia and clementincercel, respectively. firstly, we experimented with recurrent neural network (rnn) @xcite solutions alongside word embeddings. after that, we performed the leap towards transformer-based @xcite models that usually perform better and offer more insight for the combined language models. furthermore, adding an auxiliary task for training a multi-task learning (mtl) @xcite architecture can lead to even better results, as the models become able to learn new features from the input texts. the paper is structured as follows. in section 2, we perform an analysis of existing solutions for several code-mixed tasks and sentiment analysis. in section 3, we detail the proposed approaches for code-mixed sentiment analysis. section 4 details the performed experiments, including data and preprocessing, experimental setup, and a discussion of the results. finally, we draw conclusions in section 5. mtl represents a distinct approach used for training neural networks for detecting the stance of the user in code-mixed texts @xcite . the authors represented the main task as a classification with three stance classes (neutral, favor, against). at the same time, the auxiliary task is binary: either the author is neutral or has an opinion, positive or negative. their approach obtains an accuracy of 63.2% on the benchmark dataset proposed by @xcite . @xcite introduced a multi-component neural network based on both word and character representations, with the purpose of performing named entity recognition (ner) on code-switched data. the input words are defined as pre-trained fasttext @xcite embeddings. a convolutional neural network (cnn) @xcite has the role of learning characterlevel representations for words. more, another component is a bidirectional lstm (bilstm), used for learning the contextual relationships between the words in the same sentence. their performance is over the considered baseline, surpassing it by a percentage of roughly 4% in accuracy. @xcite proposed an architecture suitable for nlp tasks aiming to tackle the language identification problem in code-mixed texts, called multichannel neural network (mnn) that works alongside a bilstm-crf. the mnn module is based on the inception architecture @xcite and consists of four channels, one of them entering a lstm, while the other three being connected to 1-d convolutional networks. the convolutional layers are used for capturing word representations at the n-gram level. the bilstm-crf module is directly connected to the previous one, with the purpose of learning the probabilities of switching language mid-sentence, based on the character embeddings. after training the architecture on two code-mixed datasets (hindi-english and bengali-english), their results surpassed the accuracy of the baseline model by 5%. for performing sentiment analysis, @xcite used an english-spanish code-switching dataset containing 3,062 tweets, as well as two monolingual corpora from semeval-2014 @xcite and @xcite . at the same time, the authors used a multilingual corpus obtained by merging the previously mentioned monolingual sets. their approach for identifying the sentiment implies extracting features by considering four atomic sets: words, lemmas, psychometric properties, and part-of-speech tags. by using this solution, the authors obtained excellent results: 98.12% for the monolingual english dataset, 96.03% for spanish, as well as 98% for the multilingual dataset.