MSR I ndia at S em E val-2020 Task 9: Multilingual Models Can Do Code-Mixing Too (2020)

Abstract: In this paper, we present our system for the SemEval 2020 task on code-mixed sentiment analysis. Our system makes use of large transformer based multilingual embeddings like mBERT. Recent work has shown that these models posses the ability to solve code-mixed tasks in addition to their originally demonstrated cross-lingual abilities. We evaluate the stock versions of these models for the sentiment analysis task and also show that their performance can be improved by using unlabelled code-mixed data. Our submission (username Genius1237) achieved the second rank on the English-Hindi subtask with an F1 score of 0.726.

## introduction the task of identifying sentiment from text is extremely important in this age where large volumes of text content are being consumed via social media. the task becomes even more interesting when it comes to bilingual communities as these communities exhibit the phenomenon of code-mixing online @xcite . existing approaches to tackling this problem have mainly been based on statistical methods @xcite . these methods have used features like n-gram counts and tf-idf vectors along with a linear classifier. there have been very few approaches to this problem using deep learning as the amount of labelled code-mixed data available has always been very less. methods like the one in @xcite train word embeddings using unlabelled code-mixed data, the availability of which is not as problematic as labelled data, and use these embeddings along with a recurrent neural network based model. recent advancements in natural language processing have shown that large transformer based models like bert @xcite , when pre-trained on large corpora, are easily adaptable for downstream tasks with small datasets. these models even perform well in a cross-lingual manner @xcite when pre-trained on corpora spanning multiple languages. our experiments show that these multilingual models perform well even on code-mixing tasks, having had no exposure to any code-mixing during pre-training. we use such a system to solve the code-mixed sentiment analysis problem. we also show that it's performance can be improved by using a combination of generated and real code-mixed text. the rest of the paper is organized as follows. section 2 tasks about the dataset for the task and the pre-processing done to it. section 3 talks about the different systems we evaluated, with section 3.3 in particular going into how we improved the multilingual models using code-mixed data. section 4 describes the performance of the different models and section 5 concludes our discussion. ## dataset and preprocessing the details about the datasets @xcite for both the english-hindi (en-hi) and english-spanish (en-es) tasks are described in we make use of the language identification tool by @xcite to identify the romanized hindi sections and transliterate them to devanagari using the bing translator api foot_0 . the language tags provided along with the data is not used. no other pre-processing is done to the data. ## system description figure 1 describes the model used for sentiment analysis. the model is a classification model that comprises of a pretrained transformer-based multilingual embedding (like bert) and a linear layer acting as a classification head. the embedding takes in a tokenized sentence and outputs a single embedding for that sentence. this embedding is then run through the linear layer that outputs scores for each of the 3 classes. the entire system was implemented using the huggingface transformers library @xcite . we experimented with different models for the embedding. we also experimented with different pooling techniques that are used to obtain the sentence embedding and these are detailed below. finally, as a baseline, we report the results from the method in @xcite , using word2vec embeddings trained on code-mixed data along with a bilstm. ## results and analysis the results are presented in tables 3 and 4 . each table contains f1 scores averaged over 5 different seeds. for all the runs, a batch size of 64 was used along with the adam optimizer with a learning rate of 5e-5. each batch was made to have equal number of samples from all 3 classes. training was performed for 10 epochs. right away, we are able to observe that the stock versions of mbert and xlm-r, which are not exposed to any form of code-mixing during their pre-training show impressive f1 scores. this is talked about more in section 4.2. we present an analysis of the sentence embedding techniques first. ## conclusion in this paper, we present our system for the semeval 2020 task on code-mixed sentiment analysis. we make use of multilingual models like mbert and show that they work well for code-mixing tasks. the best performance is extracted from these models by finetuning them on code-mixed data and using this version instead of their stock versions. we also find that for simple sequence classification tasks, the choice of sentence embedding technique does not have a significant impact on the result. there are multiple paths for further exploration of this work. while finetuning mbert on code-mixed data, we've created one model per language and used a relatively small amount of data (compared to the amount of data bert is pretrained on). both these could be looked into, creating a single model for multiple language pairs, and using much more data for this purpose. in this process, one may be able to obtain a universal model that works for a large number of code-mixed pairs in addition to the large number of languages that mbert already supports.