Progressive Sentiment Analysis for Code-Switched Text Data (2022)

Abstract: Multilingual transformer language models have recently attracted much attention from researchers and are used in cross-lingual transfer learning for many NLP tasks such as text classification and named entity recognition.However, similar methods for transfer learning from monolingual text to code-switched text have not been extensively explored mainly due to the following challenges:(1) Code-switched corpus, unlike monolingual corpus, consists of more than one language and existing methods can’t be applied efficiently,(2) Code-switched corpus is usually made of resource-rich and low-resource languages and upon using multilingual pre-trained language models, the final model might bias towards resource-rich language. In this paper, we focus on code-switched sentiment analysis where we have a labelled resource-rich language dataset and unlabelled code-switched data. We propose a framework that takes the distinction between resource-rich and low-resource language into account.Instead of training on the entire code-switched corpus at once, we create buckets based on the fraction of words in the resource-rich language and progressively train from resource-rich language dominated samples to low-resource language dominated samples. Extensive experiments across multiple language pairs demonstrate that progressive training helps low-resource language dominated samples.

## introduction code-switching is the phenomena where the speaker alternates between two or more languages in a conversation. the lack of annotated data and diverse combinations of languages with which this phenomenon can be observed, makes it difficult to progress in nlp tasks on code-switched data. and also, the prevalance of different languages is different, making annotations expensive and difficult. intuitively, multilingual language models like mbert @xcite can be used for code-switched text since a single model learns multilingual representations. although the idea seems straightforward, there are multiple issues. firstly, mbert performs differently on different languages depending on their script, prevalence and predominance. mbert performs well in mediumresource to high-resource languages, but is outperformed by non-contextual subword embeddings in a low-resource setting @xcite . moreover, the performance is highly dependent on the script @xcite . secondly, pre-trained language models have only seen monolingual sentences during the unsupervised pretraining, however code-switched text contains phrases from both the languages in a single sentence as shown in figure 1 , thus making it an entirely new scenario for the language models. thirdly, there is difference in the languages based on the amount of unsupervised corpus that is used during pretraining. for e.g., mbert is trained on the wikipedia corpus. english has ∼ 6.3 million articles, whereas hindi and tamil have only ∼ 140k articles each. this may lead to under-representation of low-resource langauges in the final model. further, english has been extensively studied by nlp community over the years, making the supervised data and tools more easily accessible. thus, the model would be able to easily learn patterns present in the resourcet[cls] e1 u1 e2 u2 e3 u3 en-2 un-2 en-1 un-1 en un e[sep] label … … sentence … e[cls] t[cls] e1 u1 e2 u2 e3 u3 en-2 un-2 en-1 un-1 en un e[sep] label … … sentence … e[cls] t[cls] e1 u1 e2 u2 e3 u3 en-2 un-2 en-1 un-1 en un e[sep] label … … sentence … divide into buckets input decreasing fraction of english words final model e[cls] t[cls] e1 u1 e2 u2 e3 u3 en-2 un-2 en-1 un-1 en un e[sep] label … … sentence … e[cls] t[cls] e1 u1 e2 u2 e3 u3 en-2 un-2 en-1 un-1 en un e[sep] label … … sentence … e[cls] t[cls] e1 u1 e2 u2 e3 u3 en-2 un-2 en-1 un-1 en un e[sep] label … … sentence … e[cls] t[cls] e1 u1 e2 u2 e3 u3 en-2 un-2 en-1 un-1 en un e[sep] label … … sentence … b1 b2 b3 input m pt m 1 m 2 predict in p u t predict predict i n p u t in p u t in pu t inpu t in pu t input input t s figure 2 : a visualization of the progressive training strategy. the source labelled dataset s in resource rich language should be easily available. using s, a classifier is trained, say m pt . unlabelled code-switched dataset t is divided into buckets using the fraction of english words as the metric. the leftmost bucket b1 has samples dominated by resource-rich language and as we move towards right, the samples in the buckets are dominated by low-resource language. m pt is used to generate pseudo-labels for unlabelled texts in bucket b1. we use texts from b1 along with their pseudo-labels and the dataset s to train a second text classifier m 1 . then, m 1 is used to get the pseudo-labels for texts in bucket b2. we keep repeating this until we obtain the final model which is used for predictions. the main idea behind our paper can be summarised as follows: when doing zero shot transfer learning from a resource-rich language (langa) to code switched language (say langa-langb, where langb is a low-resource language compared to langa), the model is more likely to be wrong when the instances are dominated by langb. thus, instead of self-training on the entire corpus at once, we propose to progressively move from langa-dominated instances to langbdominated instances while transfer learning. as illustrated in figure 2 , model trained on the annotated resource-rich language dataset is used to generate pseudo-labels for code-switched data. progressive training uses the resource-rich language dataset and (unlabelled) resource-rich language dominated code-switched samples together to generate better quality pseudo-labels for (unlabelled) low-resource language dominated code-switched samples. lastly, annotated resource-rich language dataset and pseudo-labelled code-switched data are then used together for training which increases the performance of the final model. our key contributions are summarized as: • we propose a simple, novel training strategy that demonstrates superior performance. since our hypothesis is based on the pretraining phase of the multilingual language models, it can be combined with any transfer learning method. ## related work multiple tasks like language identification, named entity recognition, part-of-speech, sentiment analysis, question answering and nli have been studied in the code-switched setting. for sentiment analysis, @xcite al. (2015) showed that multilingual approaches can outperform pipelines of monolingual models on code-switched data. lal et al. (2019) use cnn based network for the same. winata et al. (2019) use hierarchical meta embed-dings to combine multilingual word, character and sub-word embeddings for the ner task. aguilar and solorio (2020) augment morphological clues to language models and uses them for transfer learning from english to code-switched data with labels. samanta et al. (2019) uses translation api to create synthetic code-switched text from english datasets and use this for transfer learning from english to code-switched text without labels in the codeswitched case. qin et al. (2020) use synthetically generated code-switched data to enhance zero-shot cross-lingual transfer learning. recently, khanuja et al. ( 2020 ) released the gluecos benchmark to study the performance of multiple models for codeswitched tasks across two language pairs en-es and en-hi. the benchmark contains 6 tasks, 11 datasets and has 8 models for every task. multilingual transformers fine tuned with masked-language-model objective on code-switched data can outperform generic multilingual transformers. results from @xcite show that sentiment analysis, question answering and nli are significantly harder than tasks like ner, pos and lid. in this work, we focus on the sentiment analysis task in the absence of labeled code-switched data using multilingual transformer models, while taking into account the distinction between resource-rich and low-resource languages. although our work seems related to curriculum learning, it is distinct from the existing work. most of the work in curriculum learning is in supervised setting @xcite and our work focuses on zero-shot setting, where no code-switched sample is annotated. note that, this is also different from semi-supervised setting because of distribution shifts between labeled resource-rich language data and target unlabeled code-switched data. ## preliminaries our problem is a sentiment analysis problem where we have a labelled resource-rich language dataset and unlabelled code-switched data. from here onwards, we refer the labelled resource-rich language dataset as the source dataset and the unlabelled code-switched dataset as target dataset. since code-switching often occurs in language pairs that include english, we refer to english as the resource-rich language. the source dataset, say s, is in english and has the text-label pairs {(x s 1 , y s 1 ), (x s 2 , y s 2 ), ...(x sm , y sm )} and the target dataset, say t , is in code-switched form and has texts {x cs 1 , x cs 2 , ...x csn }, where m is significantly greater than n. the objective is to learn a sentiment classifier to detect sentiment of codeswitched data by leveraging labelled source dataset and unlabelled target dataset. ## methodology our methodology can be broken down into three main steps: (1) source dataset pretraining, which uses the resource-rich language labelled source dataset s for training a text classifier. this classifier is used to generate pseudo-labels for the target dataset t . (2) bucket creation, which divides the unlabelled data t into buckets based on the fraction of words from resource-rich language. some buckets would contain samples that are more resourcerich language dominated while others contain samples dominated by low-resource language. (3) progressive training, where we initially train using s and the samples dominated by resource-rich language and gradually include the low-resource language dominated instances while training. for rest of the paper, pretraining refers to step 1 and training refers to the training in step 3. and, we also use class ratio based instance selection to prevent the model getting biased towards majority label. ## experiments we describe the details relevant to the experiments in this section and also elaborate on the probing tasks. ## conclusion and future work in this paper, we propose progressive training framework that takes distinction between lowresource and resource-rich language into account while doing zero-shot transfer learning for codeswitched sentiment analysis. we show that our framework improves performance across multiple datasets. further, we also create probing tasks to provide empirical evidence in support of our hypothesis. in future, we want to extend the framework to other tasks like question-answering and natural language inference.