Palomino-Ochoa at S em E val-2020 Task 9: Robust System Based on Transformer for Code-Mixed Sentiment Classification (2020)

Abstract: We present a transfer learning system to perform a mixed Spanish-English sentiment classification task. Our proposal uses the state-of-the-art language model BERT and embed it within a ULMFiT transfer learning pipeline. This combination allows us to predict the polarity detection of code-mixed (English-Spanish) tweets. Thus, among 29 submitted systems, our approach (referred to as dplominop) is ranked 4th on the Sentimix Spanglish test set of SemEval 2020 Task 9. In fact, our system yields the weighted-F1 score value of 0.755 which can be easily reproduced — the source code and implementation details are made available.

## introduction sentiment analysis is one of the most active research areas in natural language processing (nlp), web mining and social media analytics @xcite . in particular, its polarity detection task has been extensively researched since 2002 @xcite . polarity detection involves to determine whether a given text, containing an opinion, is positive, negative or neutral. in order to solve this task, several dictionary-based methods were proposed in the past @xcite . however, machine learning (ml) approaches have been the ones with more impact regarding the-state-of-the-art @xcite . moreover, traditional ml approaches based on feature engineering and modelling have been consistently improved by deep learning methods @xcite . as a rule, to apply deep learning in sentiment analysis input text should be encoded in some way, for instance using word embeddings. word embeddings are useful because allow us to encode semantic similarities among words. one can obtain a word vector representation by training a large corpus using algorithms such as word2vec @xcite , glove @xcite and fasttext @xcite , to name a few. nowadays, this kind of text representation has evolved to a language model encoding. the idea is to use language context in order to better encode words and characters. overall, the aim of this encoding is to transfer the knowledge embodied in the language model to address a specific task. thus, in this paper we focus on transfer learning: we make use of a pre-trained language model and then we applied it in sentiment classification (the aim is to predict the correct sentiment classification of a given code-mixed tweet). in order to do so, our work combines two powerful frameworks: the universal language model fine-tuning (ulmfit) @xcite and bert @xcite . ulmfit is the basis for our transfer learning strategy and it has proven to have an impressive performance on several english text classification tasks-it also has obtained optimal results for the spanish language @xcite . on the other hand, bert is currently the-state-of-the-art language model for nlp. hence, we use a reduced multilingual language model (bert) which is further fine-tuned (ulmfit) in order to classify code-mixed tweets. we evaluate our system on the sentimix spanglish test set of semeval 2020 task 9. our submission (referred to as dpalominop) obtained the weighted-f1 value of 0.755 and was ranked 4th among 29 teams. ## background the aim of the task 9 @xcite is to predict the correct sentiment classification of a given code-mixed tweet. each sentiment label can be one of three options: positive, negative and neutral. there are two code-mixed languages provided on this track: english-hindi and english-spanish. in this work we are only approached the code-mixed english-spanish task. train, evaluation and test datasets are available and contain tweets in conll-u format @xcite . thus, every tweet word is tagged accordingly: en (english), spa (spanish), hi (hindi), mixed and univ (e.g. symbols, @ mentions, hashtags, etc). the whole tweet is tagged with the corresponding sentiment label. during the competition only train and evaluation datasets were labeled. ## system overview in this section, we present the system design choices that allow us to predict the sentiment of a given code-mixed tweet. overall, our strategy is as follows, due to a small dataset is provided we plan to use a pre-trained language model. by doing so, we aim at extracting features and context words from a large corpus in order to "transfer" this knowledge to our small dataset. consequently, we only should perform a fine-tuning step regarding the task at hand. several language models have been proposed in the last years @xcite @xcite ), but the one with the highest impact in the-state-of-the-art has been bert @xcite . bert is a powerful language model that uses a bidirectional representation from unlabeled data by jointly conditioning on both left and right context words. in order to train this model, @xcite propose a novel training task called masked language model (mlm). in this task, some input tokens are randomly masked in order to predict vocabulary identifiers using context, i.e., remaining words of the sentence. in addition, the work proposes the next sentence prediction (nsp) training task that jointly pre-trains text-pairs representations. to do so, the whole text is parsed in several batches of two consecutive phrases. then, the first one is used to predict the second one. with these two training tasks, bert has outperformed previous state of the art results on several nlp challenges. transfer learning approaches based on language models for nlp tasks have been proposed in the past @xcite @xcite . arguably, the simplest proposal for performing text classification using language models is the universal language model fine-tuning (ulmfit) @xcite . such work was extended to spanish sentiment analysis with remarkable results @xcite . our proposal uses a base bert language model and embed it within a ulmfit transfer learning pipeline. the resulting system is depicted in figure 1 . figure 1: general system pipeline. w x is the input to the system and o x is the output before the pre-training task layer (mlm / nsp). the components are described as follows: so far, we have presented the main pipeline of our system, but research on bert models have produced several variants such as base, large and multilingual which rely on the number of parameters, layers and languages, accordingly. as discussed in the original paper @xcite , the best choice to use in small and medium datasets is the base version because the lower number of parameters to fine-tune. however, the code-mixed tweets aimed to classify are written in english and spanish. in this sense, the best choice would be a multilingual version but, experimental results have shown that a multilingual lm performs worst than a single lm foot_0 . regarding those results, our design choice has been to use a multilingual model with few languages (including at least english and spanish) which we referred to as reduced multilingual. in the next section we describe the pre-processing data, system configuration as well as the choice of the language model used in this challenge. ## experimental setup a complete description about hardware and software requirements for reproducing this paper is detailed in this section. in addition, we show the hyper-parameters tuned during experimentation, e.g. the learning rate that allow us to converge without overfitting and regularization. ## results results for semeval 2020 task 9 competition are reported in we have also performed some ablation experiments regarding our proposal steps so as to understand the impact of the design choices in our results. this analysis is presented in ## https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment 6 conclusion we have presented a transfer learning approach to tackle a mixed spanish-english sentiment classification task. in order to do so, we have combined a transfer learning scheme based on ulmfit with the-state-ofthe-art language model bert. that approach allowed us to be ranked 4th on the sentimix spanglish test set of semeval 2020 task 9. furthermore, we have demonstrated that a reduced multilingual language model performs better than the one supporting several languages. moreover, we have discussed the impact of a correct fine-tuning using a discriminative process on each layer regardless that some of them remain frozen during training.