Late Fusion of Transformers for Sentiment Analysis of Code-Switched Data (2023)

Abstract: Code-switching is a common phenomenon in multilingual communities and is often used on social media. However, sentiment analysis of code-switched data is a challenging yet less explored area of research. This paper aims to develop a sentiment analysis system for code-switched data. In this paper, we present a novel approach combining two transformers using logits of their output and feeding them to a neural network for classification. We show the efficacy of our approach using two benchmark datasets, viz., English-Hindi (En-Hi), and English-Spanish (En-Es) availed by Microsoft GLUECoS. Our approach results in an F1 score of 73.66% for En-Es and 61.24% for En-Hi, significantly higher than the best model reported for the GLUECoS benchmark dataset.

## introduction code-switching refers to using two or more languages in a single conversation. this linguistic feature is prevalent in multilingual communities all over the world. multilingual communities share vast amounts of unstructured code-switched data on social media in text and speech. sentiment analysis of data shared on social media has proved valuable in market predictions and analysis @xcite , mental health assessments @xcite , and intelligent agents @xcite . there have been enormous studies conducted on multilingual nlp, but, despite such prevalence of code-switching in real life, very few studies have been conducted in the field. @xcite proposed generalized language evaluation benchmark (glue) to evaluate embedding models on various language understanding tasks. inspired by this, microsoft research published gluecos @xcite , an evaluation benchmark on various tasks focused on code-switched text. sentiment analysis is one of the seven tasks present in the gluecos evaluation benchmark, which is aimed at classifying sentiment in a code-switched sentence as positive, negative, or neutral. the task consists of english-spanish and english-hindi code-switched language pairs. the english-spanish task is useful for studying the performance of models in a code-switched environment of closely related languages, and the english-hindi task is for largely different languages. the code-switched data analysis comes with its challenges. traditional techniques trained for one language (monolingual nlp models) quickly break down when input is mixed in from another language. the performance of these models goes down as the amount of mixed-language present increases @xcite . on the other hand, there is a scarcity of high-quality code-switched datasets @xcite . conversations on social media usually use slang, unknown references, mentions, improper grammar, spelling errors, etc. models trained on correct grammar and spelling will not perform well on such datasets immediately @xcite . in addition to this, there is also the case of multiple romanization. a code switch statement is usually found in the roman script because of standard english keyboards' vast popularity and availability. multiple romanization leads to words in foreign languages written in english alphabets with varying spellings, as shown in english-hindi examples in figure 1 . the cause of such different spellings is usually the difference in pronunciations which gets reflected in the absence or addition of vowels in the romanized foreign words. our contribution to the sentiment analysis of code-switched data research includes the performance improvement from the combination (late fusion) of different pre-trained models without using ensemble techniques. since pre-trained models are trained on different datasets and have different architectures, they produce varying results on the same dataset. however, a unison model, which includes multiple pre-trained models, each fine-tuned on whole training data, improves performance compared to ensemble or other voting techniques. we show that our unisom model has improved the weighted-f1 score by 10% for the english-spanish about 3.73% for the english-hindi sentiment analysis task. the rest of the paper is organized as follows. in section 2, we briefly describe the details regarding the datasets used. section 3 describes some recent related work. section 4 describes our approach. section 5 provides the experimental setup. section 6 lists the results obtained, and section 7 concludes the paper. ## dataset description gluecos is a reasonably significant evaluation benchmark for the six code-switched nlp tasks. a high performance sentiment analysis system for this dataset opens the scope for improvement in the performance of other code-switched nlp tasks. in this paper, we present results on the gluecos benchmark dataset. the dataset is generated by running a script that scrapes tweets of some presaved ids annotated beforehand @xcite . the english-spanish dataset consists of 1523 tweets for training, 184 for validation, and 232 for testing. the english-hindi dataset consists of 10080 tweets for training, 1260 for validation, and 1261 for testing. we have referred to the best results and method (mbert) reported on the gluecos website foot_0 for the task. the objective of the paper is to improve upon the state-of-the-art method, that is, mbert adapted for code-switched sentiment analysis. every data item is labeled to be expressing a positive, negative, and neutral sentiment. the detailed distribution of the annotations are listed in ## methodology due to the complementary behavior of different sentiment prediction models, we use the late fusion @xcite technique to combine results from different models as a unison model. the unison model, which includes multiple pretrained models, each fine-tuned on whole training data, improves performance compared to ensemble or other voting techniques. we combine multiple models using their logit prediction for each sentiment. the description of such a model is given in figure 2 . the logit values for each sentiment class generated from two models are fed to a neural network with three nodes in the output layer, one for each sentiment class. the neural network postmodel training handles assigning weights to the results generated by each model. it thus gives more weight to the models as and when one performs better than the other and thus produce a new model which has the better aspects of each of the contributing model. the output neuron at the output layer with the highest value determines the sentiment class for the given input. logit is described as the output of a neuron without applying the activation function. logit for a class is defined using the probability of the class being the predicted output. the mathematical definition of the logit is provided below. it is clear from the mathematical description that a larger logit value means a larger probability for the said sentiment class. ## experimental setup this section provides the experimental details to replicate the work. ## conclusion this paper presents a novel approach that uses late fusion for sentiment analysis of code-switched data. the results show that mbert-bert and xlmr-mbert fusion models perform the best for english-hindi and english-spanish sentiment analysis tasks, respectively. the tools and models used in our work are largely non-language specific and thus could be helpful in sentiment analysis of code-switched data of different languages.