Task-Specific Pre-Training and Cross Lingual Transfer for Sentiment Analysis in D ravidian Code-Switched Languages (2021)

Abstract: Sentiment analysis in Code-Mixed languages has garnered a lot of attention in recent years. It is an important task for social media monitoring and has many applications, as a large chunk of social media data is Code-Mixed. In this paper, we work on the problem of sentiment analysis for Dravidian Code-Switched languages - Tamil-Engish and Malayalam-English, using three different BERT based models. We leverage task-specific pre-training and cross-lingual transfer to improve on previously reported results, with significant improvement for the Tamil-Engish dataset. We also present a multilingual sentiment classification model that has competitive performance on both Tamil-English and Malayalam-English datasets.

## introduction code-switching is a common phenomenon which occurs in many bilingual and multilingual communities around the world. it is characterized by the usage of more than one language in a single utterance @xcite @xcite . india is one such place with many communities using different code-switched languages, with hinglish (code-switched hindi-english) being the most popular one. dravidian languages like tamil, kannada, malayalam and telugu are also usually code-mixed with english. for over 2600 years, recorded tamil literature has been documented. the 500 bce tamil script found at the site of porunthal is situated 12 km southwest of palani, tamil nadu. tamil was the first to be listed as a classical language of india, one of 22 scheduled languages in the constitution of india, and is one of the world's longestsurviving classical languages. these code-mixed languages are commonly used to interact with social media, which is why it is essential to be build systems that are able to handle code-switched data @xcite @xcite . sentiment analysis poses the task of inferring opinion and emotions of a text query (usually social media comments or tweets) as a classification problem, where each query is classified to have a positive, negative or neutral sentiment @xcite @xcite . it has various applications like understanding the sentiment of different tweets, facebook and youtube comments, understanding product reviews etc @xcite @xcite . in times of the pandemic, where the entire world is living online, it has become an even more important tool . robust systems for sentiment analysis already exist for high resource languages like english @xcite ), yet progress needs to be made for lower resource and code-switched languages. one of the major bottlenecks in dealing with code-switched languages is the lack of availability of annotated datasets in the code-mixed languages. to alleviate this problem for dravidian languages, datasets have been released for sentiment analysis in tamil-english @xcite and malayalam-english @xcite . various shared tasks @xcite have also been accompanied by the release of these datasets to advance research in this domain. models built on top of contextualized word embeddings have recently received huge amount of success and are used in most state of the art models. systems built on top of bert @xcite and its multilingual variants mbert and xlm-roberta @xcite have been the top ranking systems in both the above competitions. in this paper, we train four different bert based models for sentiment analysis for two different code-switched dravidian languages. the main con-tributions of this paper are: in our paper, we use bert, mbert @xcite , xlm-roberta @xcite and a roberta based sentiment classification model @xcite for sentiment classification. sentiment classification is usually done by classifying a query into one of three sentiments -positive, negative and neutral. in this work, we perform a three-class classification to be able to leverage the power of the tweeteval sentiment classifier @xcite , which was trained on a dataset of english tweets. the tweeteval model is a monolingual model trained on an out-of-language positive negative neutral tam-eng 10,559 2,037 850 mal-eng 2,811 738 1,903 hinglish 6,616 5,892 7,492 ## dataset in this paper, we primarily test our models on tamil-english (chakravarthi et al., 2020b) and malayalam english @xcite . the dataset was collected by scrapping youtube comments from tamil and malayalam movies. all the sentences in the dataset are in the latin script. we also use the sentimix hinglish dataset @xcite to leverage cross-lingual transfer from hinglish. the datasets statistics are summarized in ## models we train sentiment analysis models based on top of four bert variants: we use the uncased-base xlm-roberta model. we use the hugging face library implementation of these models. we expect mbert and xlm-roberta based models to leverage cross-lingual transfer from a large pool of languages it is trained on. the tweeteval model was trained on a dataset of english tweets for the task of sentiment analysis, but is still out-of domain for youtube comments datasets. it is important to note that all our chosen models either have a different domain, language or task on which they were initially trained and hence are not directly suitable for the task of codeswitched sentiment analysis of youtube comments. ## experiments in this paper, we aim to understand the effects of task-specific pre-training and cross-lingual transfer in improving performance of bert based models on code-switched datasets. task-specific pretraining refers pre-training a model on the same task for which a larger dataset is available and finetuning the so-obtained model on the target dataset, which is usually much smaller. in this case, we use the tweeteval model trained for the task of sentiment analysis on a large english corpus, which we fine-tune on the dravidian code-mixed datasets. cross-lingual transfer is commonly referred to the phenomenon of leveraging features and contextual information learnt from a different set of languages for a task with a new target language. leveraging cross-lingual transfer is one of the main reasons behind training multilingual bert models, where we expect multilingual bert models to perform better on an unknown language when compared to an english bert model. ## conclusion in this paper we present various experiments to compare the effects of task-specific pre-training and cross-lingual transfer on performance of sentiment classification models for code-switched data. to do so, we check the performance of four bert models on two code-switched languages -malaylam-english and tamil english. we find that task-specific pre-training is superior to cross-lingual transfer for our chosen code-switched datasets. the results presented in this paper for four different bert models can be used as baselines for future work on sentiment analysis for the chosen datasets. we also present first results for the tweet-eval @xcite sentiment classification model for code-switched data. in dravidian languages. in proceedings of the first workshop on speech and language technologies for dravidian languages. association for computational linguistics.