XLP at S em E val-2020 Task 9: Cross-lingual Models with Focal Loss for Sentiment Analysis of Code-Mixing Language (2020)

Abstract: In this paper, we present an approach for sentiment analysis in code-mixed language on twitter defined in SemEval-2020 Task 9. Our team (referred as LiangZhao) employ different multilingual models with weighted loss focused on complexity of code-mixing in sentence, in which the best model achieved f1-score of 0.806 and ranked 1st of subtask- Sentimix Spanglish. The performance of method is analyzed and each component of our architecture is demonstrated.

## introduction sentiment analysis is in the area of research that perform the automatic comprehension of the subjective information from user-generated data, which helps to gain the views on certain topics. due to the rise of social media such as micro-blogs (e.g., twitter) and the trend of global communications, they have accelerated the use of multilingual expressions, raising the concerns on code-mixing behavior @xcite . to develop cross-lingual encoders that can encode any sentence into a shared embedding space, by using monolingual transfer learning, multilingual extensions of pretrained @xcite encoders have been shown effective. as for code-mixed text, more complicated than cross-lingual sentence, it is crucial to consider the complexity of texts written in several different languages because different types of integration correlate with different social contexts @xcite . sometimes, the user may post blogs in non-native language with grammar mistakes or even prefer to express the sentiment in the native language. the phenomena has encouraged the researchers to analyze the sentiment from multilingual code-mixed texts. because spanish and english share a lot of words with latin roots, sometimes words with the same origin take a separate path in each language, or words with different origins resemble each other by coincidence, but have different meanings. for example, éxito from spanish means success, which resembles exit from english, with different meaning and sentiment. in the task, the number of words in a sentence vary from different languages dramatically. intuitively, the language that has a bigger presence in the tweet would contain the sentiment of the sentence. to tackle the problem, we adopt the focal loss through calculating the ratio of each language in code-mixing text. the rest of the paper is structured as follows: section 2 provides the detailed implementation method. section 3 presents the results and performance of our models as well as experiment settings. concluded remarks and future directions of our work are summarized in section 4. first, user name mentioned and url are all removed because they are useless for sentiment prediction. special characters like "rt" representing re-tweet is also deleted. moreover, we also remove the hash symbol from hash-tags as it can be problematic for tokenizers to work with. as for non-text symbol like emoji and emoticon, we use the (emoji, 2019) library from python and emoticon dictionary from wiki (list of emoticons, 2020) respectively to transform the symbols to text. next all characters into lowercase and stop words are removed. afterwords, we employ fastbpe to generate and apply bpe codes to get post-bpe vocabulary using vocabulary of xlm model for 100 languages including hindi, spanish, and english. sentence size is limited to 256. this is enough for nearly all of the tweets after processing. ## conclusion in this paper, we have introduced a novel approach with weighted loss of different multilingual models with weighted loss focused on complexity of code-mixing sentences for sentiment analysis task in semeval-2020. the method is effective in situation where the distribution of different languages is unbalanced, and has a better control of language preference for sentiment by the level of how languages mix. moreover, we conclude that the quality of word representations used has a significant impact on the performance of a model. results indicate the potency of xlm on code-mixed lingual classification, leading to 4-5 % increase in f1 score compared to muse. in the future, we will continue to do model optimization and also try ensemble models.