F i SSA at S em E val-2020 Task 9: Fine-tuned for Feelings (2020)

Abstract: In this paper, we present our approach for sentiment classification on Spanish-English code-mixed social media data in the SemEval-2020 Task 9. We investigate performance of various pre-trained Transformer models by using different fine-tuning strategies. We explore both monolingual and multilingual models with the standard fine-tuning method. Additionally, we propose a custom model that we fine-tune in two steps: once with a language modeling objective, and once with a task-specific objective. Although two-step fine-tuning improves sentiment classification performance over the base model, the large multilingual XLM-RoBERTa model achieves best weighted F1-score with 0.537 on development data and 0.739 on test data. With this score, our team jupitter placed tenth overall in the competition.

## introduction code-mixing is a phenomenon in which two or more languages are used in a single utterance. it occurs at various levels of linguistic structure: across sentences (i.e., inter-sentential), within a sentence (i.e.,intrasentential), or at the word/morpheme level. in addition to spoken language, this phenomenon has become especially prevalent on social media. as monolingual systems cannot deal with the code-mixed data, it poses a major challenge for even the most standard nlp tasks. to this end, semeval 2020 task 9 @xcite proposes the sentiment analysis task for code-mixed social media text, specifically on english-spanish (spanglish) and english-hindi (hinglish) language pairs. in this paper, we present our approach called fine-tuned spanglish sentiment analysis, or fissa for short. we focus on various pre-trained language models for spanglish sentiment classification by fine-tuning their contextualized word embeddings. by doing so, we examine two challenging aspects of code-mixed language processing: (a) multilinguality, (b) domain. we firstly compare monolingual models with their multilingual counterparts to evaluate the multilingual solution on code-switching data, considering the first aspect. secondly, to see the domain effect, we further fine-tune the multilingual model on domain-specific unlabeled data. finally, we use the most recent state-of-the-art pre-trained model and compare it to our custom model with domain information. our research shows that fine-tuning a pre-trained language model is a good choice compared to the standard blstm model when training data is limited. however, on code-mixed data, monolingual pretrained models tend to perform better on different portions of the data depending on the use of languages. as the best alternative, a large multilingual model provides better generalization and results in a stronger performance. on code-switched data including spanish-english (spa-en), many systems benefited from a combination of machine learning methods such as an svm, a crf, an extended markov model, and hand crafted or frequency-based features. on spa-en, the best performing system employed a crf classifier by using various character-and word-level features together with external resources that include monolingual corpora with named-entity lists @xcite . with regard to english-spanish sentiment analysis on social media data, the first english-spanish code-switching twitter corpus annotated with sentiment labels was made available in the research conducted by @xcite and @xcite . in their trinary annotated corpus, a collection of 3062 tweets were annotated by three annotators fluent in both english and spanish, classifying each tweet as either positive, neutral or negative. they discovered that there was a small advantage to be gained from using a multilingual approach. however, both monolingual and multilingual approaches struggled with code-switching text. pre-trained transformers deep, transformer @xcite based language models provide general-purpose contextualized linguistics representations that have shown great success on various nlp tasks @xcite @xcite . these models are pre-trained on large unannotated corpora, and then fine-tuned for downstream tasks according to task-specific objectives. as well as monolingual transformer models, multilingual models that are pre-trained on the concatenation of monolingual corpora from multiple languages, have enabled significant advances in multilingual nlp @xcite @xcite . @xcite showed that multilingual bert @xcite provides a strong cross-lingual generalization, which allows for the incorporation of information from multiple languages, for example in a code-switching scenario. ## results the results of each model are presented in our custom model (custom bert-base) for which we used a two-step fine-tuning strategy, once with a language modeling objective to inject domain information and once for task-specific fine-tuning, clearly performed better than multilingual bert. this shows that even very small amount of domain-specific, code-mixed data improves language model quality, when used for further training. however, it still could not match the performance of xlm-roberta model which is trained on larger corpora and has a larger transformer network. ## discussion to better understand our results, we performed some error analysis. since our data consisted of tweets with both english and spanish words, we expected the multilingual models to perform better than their monolingual counterparts. however, we did not see this pattern in our results. multilingual bert model performed better than the english bert, but worse than the spanish one. to see in which regard our models differ from each other, we investigated the differences in tokenization, and the effect of language use. ## conclusion in this paper we presented fissa, our approach for sentiment classification on spanish-english data. we showed that fine-tuning a pre-trained language model is a good alternative to a standard model, especially when the amount of labeled training data is limited. by fine-tuning xlm-roberta, we achieved a weighted f1-score of 0.537 on development data and 0.739 on test data in the discussion, we evaluated the effect of tokenization and language-specific performance of each model to better understand the overall results.