Classification of Code-Mixed Text Using Capsule Networks (2021)

Abstract: A major challenge in analysing social me-dia data belonging to languages that use non-English script is its code-mixed nature. Recentresearch has presented state-of-the-art contex-tual embedding models (both monolingual s.a.BERT and multilingual s.a.XLM-R) as apromising approach. In this paper, we showthat the performance of such embedding mod-els depends on multiple factors, such as thelevel of code-mixing in the dataset, and thesize of the training dataset. We empiricallyshow that a newly introduced Capsule+biGRUclassifier could outperform a classifier built onthe English-BERT as well as XLM-R just witha training dataset of about 6500 samples forthe Sinhala-English code-mixed data.

## introduction social media has become very popular among people across the world during the last decade, mainly due to the popularity of smart mobile phones. for low-resource languages, this social media data has become a major source of text in building natural language processing (nlp) applications. most of the content in social media tends to be informal. when the users are (at least to a certain degree) multilingual, the informal content they publish in social media tends to be code-mixed. code-mixed data is a result of code-switching, which denotes a shift from one language to another within a single utterance @xcite . recent work on text classification with code-mixed data has used contextual embedding models such as bert @xcite , and their multilingual versions, such as mbert or xlm-r @xcite . however, using these pre-trained models for code-mixed text classification, in particular the domain-specific data in low-resource languages poses many challenges. contextual embedding models such as bert need large volumes of monolingual data to train. on the other hand, not every language is included in the pre-trained multilingual models, and low-resource languages are underrepresented in those models (due to the smaller amounts of low-resource language data used in contrast to high-resource languages when training these models). in fact, there is a line of research that has shown even simple classifiers such as logistic regression proving to be more effective than the multilingual embedding models @xcite . in this paper, we empirically show that the success of contextual embedding models on code-mixed text classification depends on multiple factors, and they can indeed be sub-optimal compared to text classification based on neural models other than transformer based ones. we selected sinhala and malayalam, which are low-resource languages. in the recent language categorization by @xcite , sinhala belongs to class 0 (i.e. it has exceptionally limited resources). malayalam is categorised as 1, (i.e. it has some unlabelled data, however collecting labelled data is challenging). a food recipe dataset with about 3000 samples @xcite was used as the malayalam-english code-mixed data. for sinhala-english code-mixed data, a corpus of 10000 user comments in the domain of telecommunication was annotated with two types of information: aspects related to the telecommunication domain, and overall sentiment of the comment. we fine-tuned english-bert and xlm-r (conneau et al., 2019) models on both datasets. then we implemented a novel capsule+bigru network for the same tasks. results show that the capsule+bigru model consistently outperforms english-bert and xlm-r models for the sinhala-english dataset that had more data and less codemixing complexity than the malayalam-english dataset. with this, we establish the argument that the performance of contextual embedding mod-els depends on multiple factors such as the codemixing level, size of the dataset used to train the contextual embedding models, and the size of the dataset used in fine-tuning. further experiments with the sinhala-english dataset showed that this capsule+bigru model is superior to recurrent models as well. the annotated dataset, as well as our code are publicly released foot_0 2 related work ## conclusion the objective of this research was to critically analyse the performance of english-bert and xlm-r models for classifying code-mixed data. we identified that the performance of these models depends on factors such as the size and composition of the code-mixed data. we were able to introduce a novel capsule+bigru model that could outperform the english-bert and xlm-r models with a moderate dataset of sinhala-english 10000 comments (note in 3-fold cross-validation, about 6600 samples are used for training). this result suggests that, at least for text classification on code-mixed data that involves extremely low-resource languages that are under-represented in the large multilingual embedding models, traditional deep learning solutions are still a viability. this research can be considered as one of the very few works that comparatively analysed the performance of these different techniques for code-mixed data with respect to multiple factors and languages. furthermore, this research publicly released a code-mixed dataset that can be used for two text classification tasks for the extremely low resource language sinhala. we believe that further research should be conducted with respect to more languages in order to properly determine the impact of the aforementioned factors on classification of code-mixed text.