Unsupervised Self-Training for Sentiment Analysis of Code-Switched Data (2021)

Abstract: Sentiment analysis is an important task in understanding social media content like customer reviews, Twitter and Facebook feeds etc. In multilingual communities around the world, a large amount of social media text is characterized by the presence of Code-Switching. Thus, it has become important to build models that can handle code-switched data. However, annotated code-switched data is scarce and there is a need for unsupervised models and algorithms. We propose a general framework called Unsupervised Self-Training and show its applications for the specific use case of sentiment analysis of code-switched data. We use the power of pre-trained BERT models for initialization and fine-tune them in an unsupervised manner, only using pseudo labels produced by zero-shot transfer. We test our algorithm on multiple code-switched languages and provide a detailed analysis of the learning dynamics of the algorithm with the aim of answering the question - ‘Does our unsupervised model understand the Code-Switched languages or does it just learn its representations?’. Our unsupervised models compete well with their supervised counterparts, with their performance reaching within 1-7% (weighted F1 scores) when compared to supervised models trained for a two class problem.

## introduction sentiment analysis, sometimes also known as opinion mining, aims to understand and classify the opinion, attitude and emotions of a user based on a text query. sentiment analysis has many applications including understanding product reviews, social media monitoring, brand monitoring, reputation management etc. code switching is referred to as the phenomenon of alternation between multiple languages, usually two, within a single utterance. code switching is very common in many bilingual and multilingual societies around the world including india (hinglish, tanglish etc.), singapore (chinglish) and various spanish speaking areas of north america (spanglish). a large amount of social media text in these regions is code-mixed, which is why it is essential to build systems that are able to handle code switching. various datasets have been released to aid advancements in sentiment analysis of code-mixed data. these datasets are usually much smaller and more noisy when compared to their high-resourcelanguage-counterparts and are available for very few languages. thus, there is a need to come up with both unsupervised and semi-supervised algorithms to deal with code-mixed data. in our work, we present a general framework called unsupervised self-training algorithm for doing sentiment analysis of code-mixed data in an unsupervised manner. we present results for four code-mixed languages -hinglish (hindi-english), spanglish (spanish-english), tanglish (tamil-english) and malayalam-english. in this paper, we propose the unsupervised self-training framework and apply it to the problem of sentiment classification. our proposed framework performs two tasks simultaneously -firstly, it gives sentiment labels to sentences of a codemixed dataset in an unsupervised manner, and secondly, it trains a sentiment classification model in a purely unsupervised manner. the framework can be extended to incorporate active learning almost seamlessly. we present a rigorous analysis of the learning dynamics of our unsupervised model and try to answer the question -'does the unsupervised model understand the code-switched languages or does it just recognize its representations?'. we also show methods for optimizing performance of the unsupervised self-training algorithm. ## related work in this paper, we propose a framework called unsupervised self-training, which is an extension to the semi-supervised machine learning algorithm called @xcite . self-training has previously been used in natural language processing for pre-training @xcite and for tasks like word sense disambiguation @xcite . it has been shown to be very effective for natural language processing tasks @xcite and better than pre training in low resource scenarios both theoretically @xcite and emperically @xcite . @xcite show that selftraining can be a more useful than pre-training in high resourced scenarios for the task of object detection, and a combination of pre-training and selftraining can improve performance when only 20% of the available dataset was used. however, our proposed framework differs from self-training such that we only use the zero-shot predictions made by our initialization model to train our models and never use actual labels. sentiment analysis is a popular task in industry as well as within the research community, used in analysing the markets @xcite , election campaigns @xcite etc. a large amount of social media text in most bilingual communities is code-mixed and many labelled datasets have been released to perform sentiment analysis. we will be working with four code-mixed datasets for sentiment analysis, malayalam-english and tamil-english @xcite and spanglish and hinglish @xcite . previous work has shown bert based models to achieve state of the art performance for codeswitched languages in tasks like offensive language identification (jayanthi and @xcite and sentiment analysis @xcite . we will build unsupervised models on top of bert. bert @xcite based models have achieved state of the art performance in many downstream tasks due to their superior contextualized representations of language, providing true bidirectional context to word embeddings. we will use the sentiment analysis model from @xcite , trained on a large corpus of english tweets (60 million tweets) for initializing our algorithm. we will refer to the sentiment analysis model from @xcite as the tweeteval model in the remainder of the paper. the tweeteval model is built on top of an english roberta (liu et al., 2019) model. ## proposed approach: unsupervised self-training our proposed algorithm is centred around the idea of creating an unsupervised learning algorithm that is able to harness the power of cross-lingual transfer in the most efficient way possible, with the aim of producing unsupervised sentiment labels. in its most fundamental form, our proposed unsupervised self-training algorithm foot_0 is shown in figure 1 is shown in figure 1 . we begin by producing zero-shot results for sentiment classification using a selected pre-trained model trained for the same task. from the predictions made, we select the top-n most confident predictions made by the model. the confidence level is judged by the softmax scores. making the zero-shot predictions and selecting sentences make up the initialization block as shown in figure 1 . we then use the pseduo-labels predicted by the zero-shot model to fine tune our model. after that, predictions are made on the remaining dataset with the fine-tuned model. we again select sentences based on their softmax scores for fine-tuning the model in the next iteration. these steps are repeated until we've gone through the entire dataset or until a stopping condition. at all fine-tuning steps, we only use the predicted pseduo-labels as ground truth to train the model, which makes the algorithm completely unsupervised. as the first set of predicted pseudo-labels are produced by a zero-shot model, our framework is very sensitive to initialization. care must be taken to initialize the algorithm with a compatible model. for example, for the task of sentiment classification of hinglish twitter data, an example of a compatible initial model would be a sentiment classification model trained on either english or hindi sentiment data. it would be even more compatible if the model was trained on twitter sentiment data, the data thus being from the same domain. ## datasets we test our proposed framework on four different languages -hinglish @xcite , spanglish @xcite , tanglish @xcite and @xcite . the statistics of the training sets are given in the choice of datasets, apart from covering three language families, incorporate several other important features. we can see from the datasets also differ in the amount of classwise code-mixing. figure 3 shows that for the hinglish dataset, a negative tweet is more likely to contain large amounts of hindi. this is not the same for the other datasets. for spanglish, both positive and negative sentiment tweets have a tendency to use a larger amount of spanish than english. an important thing to note here is that each of the four code-mixed datasets selected are written in the latin script. thus our choice of datasets does not take into account mixing of different scripts. ## models models built on top of bert @xcite roberta @xcite have recently produced state-of-the-art results in many natural language processing tasks. various shared tasks @xcite in the domain of code-switched sentiment analysis have also seen their best performing systems build on top of these bert models. english is a common language among all the four code-mixed datasets being considered. this is why we use a roberta based sentiment classification model trained on a large corpus of 60 million english tweets @xcite for initialization. we refer to this sentiment classification model as the tweeteval model for the rest of this paper. we use the hugging face implementation of the tweeteval sentiment classification model 2 . the models are fine-tuned with a batch size of 16 and a learning rate of 2e-5. the tweeteval model preprocesses sentiment data to not include any url's. we have done the same for for all the four datasets. we compare our unsupervised model with a set of supervised models trained on each of the four datasets. we train supervised models by fine tuning the tweeteval model on each of the four datasets. our experiments have shown that the tweeteval model performs the better in comparison to mbert and xlm-roberta based models for code-switched data. ## evaluation we evaluate our results based on weighted average f1 and accuracy scores. when calculating the weighted average, the f1 scores are calculated for each class and a weighted average is taken based on the number of samples in each class. this metric is chosen because three out of four datasets we work with are highly imbalanced. we use the sklearn implementation for calculating weighted average f1 scores 3 . there are two ways to evaluate the performance of our proposed method, corresponding to two different perspectives with which we look at the outcome. one of the ways to evaluate the proposed method is to answer the question -'how good is the model when trained in the proposed, unsupervised manner?'. we call this perspective of evaluation, having a model perspective. here we're evaluating the strength of the unsupervised model. to evaluate the method from a model perspective, we compare the performance of best unsupervised model with the performance of the supervised model on the test set. the second way to evaluate the proposed method is by looking at it from what we call an algorithmic perspective. the aim of proposing an unsupervised algorithm is to be able to select sentences belonging to a particular sentiment class from an unkown dataset. hence, to evaluate from an algorithmic perspective, we must look at the training set and check how accurate the algorithm is in its annotations for each class. to do this, we show performance (f1-scores) as a function of the number of selected sentences from the training set. ## experiments for our experiments, we restrict the dataset to consist of two sentiment classes -positive and negative sentiments. in this section, we evaluate our proposed unsupervised self-training framework for four different code-swithced languages spanning across three language families, with different dataset sizes and different extents of imbalance and code-mixing, and across two different domains. we also present a comparison between supervised and unsupervised models. we present two training strategies for our proposed unsupervised self-training algorithm. the first is a vanilla strategy where the same number of sentences are selected in the selection block for each class. the second strategy uses a selection ratio -where we select sentences for fine tuning in a particular ratio from each class. we evaluate the algorithm based on the two evaluation criterion described in section 6. in the fine-tune block in figure 1 , we fine-tune the tweeteval model on the selected sentences for only 1 epoch. we do this because we do not want our model to overfit on the small amount of selected sentences. this means that when we go through the entire training dataset, the model has seen every sentence in the train set exactly once. we see that if we fine-tune the model for multiple epochs, the model overfits and its performance and capacity to learn reduces with every iteration. ## analysis in this section we aim to understand the information learnt by a model trained under the unsupervised self-training. we take the example of the hinglish dataset. to do so, we define a quantity called token ratio to quantify the amount of code-mixing in a sentence. since our initialization model is trained on an english dataset, the language of interest is hindi and we would like to understand how well our model handles sentences with a large amount of hindi. hence, for the hinglish dataset, we define the hindi token ratio as: hindi token @xmath0, eng, 0, where 0 usually corresponds to a symbol or other special characters in a tweet. to quantify amount of code-mixing, we only use the tokens that have eng or hin as labels. words are defined as tokens that have either the label hin or eng. we define the token ratio quantity with respect to hindi, but our analysis can be generalized to any code-mixed language. the distribution of the hindi token ratio (htr) in the hinglish dataset is shown in figure 3 . the figure clearly shows that the dataset is dominated by tweets that have a larger amount hindi words than english words. this is also true for the other three datasets. ## conclusion we propose the unsupervised self-training framework and show results for unsupervised sentiment classification of code-switched data. the algorithm is comprehensively tested for four very different code-mixed languages -hinglish, spanglish, tanglish and malayalam-english, covering many variations including differences in language families, domains, dataset sizes and dataset imbalances. the unsupervised models performed competitively when compared to supervised models. we also present training strategies to optimize the performance of our proposed framework. an extensive analysis is provided describing the learning dynamics of the algorithm. the algorithm is initialized with a model trained on an english dataset and has poor zero-shot performance on sentence with large amounts of code-mixing. we show that with every iteration, the performance on finetuned model increases for sentences with a larger amounts of code-mixing. eventually, the model begins to understand the code-mixed data.