Transfer Learning for Code-Mixed Data: Do Pretraining Languages Matter? (2023)

Abstract: Monolinguals make up a minority of the world’s speakers, and yet most language technologies lag behind in handling linguistic behaviours produced by bilingual and multilingual speakers. A commonly observed phenomenon in such communities is code-mixing, which is prevalent on social media, and thus requires attention in NLP research. In this work, we look into the ability of pretrained language models to handle code-mixed data, with a focus on the impact of languages present in pretraining on the downstream performance of the model as measured on the task of sentiment analysis. Ultimately, we find that the pretraining language has little effect on performance when the model sees code-mixed data during downstream finetuning. We also evaluate the models on code-mixed data in a zero-shot setting, after task-specific finetuning on a monolingual dataset. We find that this brings out differences in model performance that can be attributed to the pretraining languages. We present a thorough analysis of these findings that also looks at model performance based on the composition of participating languages in the code-mixed datasets.

## introduction in multilingual societies, contact between multiple languages has resulted in a plethora of linguistic phenomena that have long been the subject of study in linguistics, and more recently in nlp. one such phenomenon is code-switching, or code-mixing foot_0 , in which speakers use material from two or more languages within the same conversation @xcite . code-mixing typically occurs in informal registers and casual conversations, permitted or constrained by different sociolinguistic factors @xcite . the typical lack of formality surrounding the use of code-mixing contributes to difficulties in data collection, as code-mixing is less likely to occur in official documents by governments and organizations, which have been reliable resources for the creation of many datasets @xcite . in contrast, social media has been a particularly fruitful domain for sourcing code-mixed data, useful in a wide variety of downstream tasks @xcite @xcite @xcite . among these tasks, sentiment analysis and offensive language detection stand out in particular, as @xcite have demonstrated that multilingual speakers are likely to utilize code-mixing to express their emotions, especially when cursing. thus, improving methodologies for working with intricate code-mixed data is highly relevant to the study of sentiment analysis, and social media at large. the advent of pretrained language models (plms) has tangibly shaped the norms for working with most languages, yet the implications for codemixed data are much less clear. plms have so far largely operated under monolingual assumptions and biases @xcite . most plms, including the massively multilingual ones, are trained on large web corpora, and studies have shown that the quality filters and data selection methodologies for these data sources tend to exclude text with dialectal nuances, such as text with non-standard varieties of english like african american english, or hispanic-aligned english. @xcite . attempts have been made at language modelling for code-mixed data @xcite , but an interesting question remains about how much the languages used in the pretraining of plms interact with each other to impact their performance on code-mixed data. a better understanding of this would enable targeted resource allocation to code-mixed nlp, and also potentially help understand how plms process language. plms that have been pretrained on many high-and lowresource languages are now widely available and accessible, which provides a fertile ground for such analyses @xcite . to shape the focus of this study, we introduce our hypothesis below. hypothesis: plms trained exclusively on data from relevant languages would demonstrate better performance than those that contain other extraneous languages and/or are only trained on one language. at the same time, the "curse of multilinguality", coined by @xcite , refers to the trade-off between adding more languages to increase cross-lingual capabilities, and the consequences of adding too many which can ultimately lead to loss of performance across the board in monolingual and cross-lingual benchmarks. massively multilingual models can be susceptible to this, and therefore we presume that models trained on data from relevant language families would be at an advantage. to this end, we test the performance of 7 pretrained language models on the task of sentiment analysis for different code-mixed datasets, which cover 6 languages. ## languages and datasets the datasets used in this study are mainly comprised of text scraped from twitter, facebook and youtube. details are summarised in ## experiments we used the massive choice ample tasks (machamp) (van der goot et al., 2021) codebase for the experiments. machamp provides an efficient and effective way to finetune plms on downstream tasks. ## analysis it can be inferred from the above results that for code-mixed datasets, when finetuning a plm on the code-mixed language, the languages seen in the pretraining may not substantially impact the ## summary we summarise the findings of the paper in this section to answer the main underlying question of this work -do pretraining languages matter? we approach this question for code-mixed data in two transfer learning settings: with in-language finetuning, and zero-shot.