Deep Learning Brasil - NLP at S em E val-2020 Task 9: Sentiment Analysis of Code-Mixed Tweets Using Ensemble of Language Models (2020)

Abstract: In this paper, we describe a methodology to predict sentiment in code-mixed tweets (hindi-english). Our team called verissimo.manoel in CodaLab developed an approach based on an ensemble of four models (MultiFiT, BERT, ALBERT, and XLNET). The final classification algorithm was an ensemble of some predictions of all softmax values from these four models. This architecture was used and evaluated in the context of the SemEval 2020 challenge (task 9), and our system got 72.7% on the F1 score.

## introduction it is a common tendency among multilingual people who are non-native english speakers to code-mix in their speech using english-based phonetic typing. this linguistic phenomenon, particularly in social media like twitter foot_1 , poses a great challenge to the conventional natural language processing (nlp) study area. within the context of the sentiment analysis, the study of the phenomenon of code-mixed language is important to the research community because this behavior is more common today. the interest in this area has grown due to the volume of data that social networks generate, and also by the value that this information has to understand people opinions when they are expressed in written texts. in this paper, we explain our methodology to predict sentiment in tweets, describing how our method is based on a combination of the latest language models, and also how such models contributed to a great advance in this task. this configuration was employed and evaluated in the semeval 2020 challenge (task 9), in which the goal is to predict the sentiment in code-mixed texts written in english and hindi languages of a tweet @xcite . the models used in this combination are: multifit @xcite that an evolution of ulmfit @xcite , bert @xcite , albert @xcite and xlnet @xcite . this work is organized as follows: section 2 explains some related works, section 4 describes the dataset used, section 3 addresses the methodology applied in the task, section 5 presents the results, and finally section 6 expose our final considerations as well as possible future works. ## related works sentiment analysis in twitter has been considered as a very important task from various academic and commercial perspectives. many companies use the data on twitter to decide about marketing and business decisions. a challenge is to apply sentiment analysis in texts written in two different languages like english and hindi. this happens because some people around the world speak two or more languages and sometimes when these people write texts they write it in more than one language. the author @xcite present their work for sentiment analysis for indian languages (sail) (code mixed). they implemented an algorithm using multinomial naïve bayes trained using n-gram and sentiwordnet features. for @xcite , users tend to express their thoughts by mixing words from multiple languages, because in most of the time, they are comfortable in their native language, and mixed languages are common when users write in social media. they divide their technique into two stages, viz language identification, and sentiment mining approach. they evaluated their results and compared to a baseline obtained from machine-translated sentences in english, and found it to be around 8% better in terms of precision. for @xcite is very important a preprocessing step to remove noise from raw text. the authors developed a multilayer perceptron to determine sentiment polarity in code-mixed social media text from facebook. such example using texts from facebook is important to show that this phenomenon can be applied in different social media than the one used on semeval 2020. @xcite cites some data about the census in india, showing the existence of 22 scheduled languages and 462 million users on the internet. to express their feelings, such users probably use more than one language to write text on social media. in order to help nlp researchers, a corpus was created by @xcite using english-hindi code-mixed of tweets marked for the presence of sarcasm and irony where each token is also annotated with a language tag. in this present work, this corpus was used to training multifit model. ## methodology the methodology applied in this task consists of training and using prediction values of four models: multifit, bert, albert, and xlnet. after retrieving prediction values, our ensemble calculates an average of all softmax values from these four models, as shown in figure 1 . models bert, albert, and xlnet were trained on a dgx-1, while multifit was trained on a gtx 1070 ti 8gb. the hyperparameters of the four models are described in ## conclusion in this paper, we propose a combination of four models for the semeval 2020 (task 9), and our team got 72.7% on f1 score in the competition. all of these models are based on using language models and transfer learning. they alone performed well, but together in an ensemble combination, they performed even better. in some applications, it is difficult to use an ensemble consisted of four models, especially because of the overhead coming from time spent on inference, culminating in an approach that sometimes will not perform well. on the other hand, the individual results of these four models are very close, meaning that for this task, any model can be used. it is important to note that multfit has the worst result, but the difference is very small, and this specific model takes a lot less time to train, being the lightest model of the ensemble. as future works, we intend to explore these models for sentiment analysis in other multilingual and monolingual scenarios.