BRCC and S enti B ahasa R ojak: The First B ahasa Rojak Corpus for Pretraining and Sentiment Analysis Dataset (2022)

Abstract: Code-mixing refers to the mixed use of multiple languages. It is prevalent in multilingual societies and is also one of the most challenging natural language processing tasks. In this paper, we study Bahasa Rojak, a dialect popular in Malaysia that consists of English, Malay, and Chinese. Aiming to establish a model to deal with the code-mixing phenomena of Bahasa Rojak, we use data augmentation to automatically construct the first Bahasa Rojak corpus for pre-training language models, which we name the Bahasa Rojak Crawled Corpus (BRCC). We also develop a new pre-trained model called “Mixed XLM”. The model can tag the language of the input token automatically to process code-mixing input. Finally, to test the effectiveness of the Mixed XLM model pre-trained on BRCC for social media scenarios where code-mixing is found frequently, we compile a new Bahasa Rojak sentiment analysis dataset, SentiBahasaRojak, with a Kappa value of 0.77.

## introduction code-mixing is common in multilingual societies @xcite . people tend to use one primary language for grammar and scripting @xcite , and other languages as auxiliary. code-mixing is commonly found on social media, such as facebook, twitter, or any other microblog services. malaysia reflects a multilingual society that considers malay to be the national language but uses mixed languages in daily life. bahasa rojak @xcite is one of the code-mixing examples that combines malay and english into a certain level of language structure. bahasa rojak or malaysian english @xcite is often mixed with chinese because the ethnic chinese population in malaysia is quite large. @xcite show that many users in malaysia use bahasa rojak on facebook. we can find many code-mixing combinations like malay-english, hindi-english, spanish-english, and others on social media. people not only express their feelings on social media, but also exchange information such as the latest news from their hometowns or countries, and financial topics are also popular. therefore, natural language processing (nlp) studies are increasingly focused on code-mixing @xcite . one of the most advanced trends in natural language processing is to prepare a large unlabeled corpus to pre-train a language model for representing the input text. the source of this unlabeled corpus is usually wikipedia @xcite . however, wikipedia does not have any bahasa rojak pages, which hinders the training of a pre-trained model that can represent bahasa rojak input texts. as a result, in this study, we employ data augmentation to automatically construct a new bahasa rojak code-mixing corpus, called brcc, for pretraining language models. to find the best way of exploiting the corpus, we not only pre-train language models including bert and xlm on brcc, but also revise the original xlm model to make it able to handle code-mixing input text. the revised model is called mixed xlm. as we mentioned, bahasa rojak is most frequently used in social media texts. hence, we compile the first bahasa rojak sentiment analysis dataset, called sen-tibahasarojak, to evaluate each language model's performance and reflect its ability to represent bahasa rojak input texts. ## corpus compilation in this study, we mainly deal with three languages: english, malay, and bahasa rojak. for each language, we construct a corpus to pre-train each language model and a sentiment analysis dataset to evaluate each language model's performance. the reason for choosing sentiment analysis is that the frequency of code-mixing on social media texts is relatively high @xcite . we will continue to discuss more details about the reason for choosing sentiment analysis in 3.4. ## experiments and results in this section, we will explain our proposed method for dealing with code-mixing data. bert and xlm are chosen as the baseline language models for comparison, and use a variety of sentiment analysis task datasets to evaluate each model. we use the movie reviews, product reviews, and stock market comments described in the previous section as the experimental datasets. we will also report the accuracy and f1 score of each model for each dataset. ## conclusion in this paper, for bahasa rojak, we build a corpus called brcc to pre-train bahasa rojak's language model, and compile a sentiment analysis dataset called sentibahasarojak. brcc and sen-tibahasarojak are the first resources available for bahasa rojak in this area. we also propose a new pretrained model, mixed xlm, which not only achieves the best performance on code-mixing data, but also maintains performance on monolingual data. our new bahasa rojak corpus is generated by our new data augmentation algorithm that recognizes three types of phrases in sentences and randomly selects some of those three phrases for translation to generate bahasa rojak sentences. our proposed mixed xlm model is able to la-bel input tokens to deal with code-mixing phenomena. as long as the mixed xlm model is pretrained on a code-mixing corpus, it can be used in downstream tasks containing code-mixing sentences, just as in this study, the mixed xlm model was used in bahasa rojak's sentiment analysis. finally, we evaluate the mixed xlm model pretrained on brcc through the sentiment analysis task on three different language settings (english, malay, bahasa rojak). the sentiment analysis task includes three domains. the results show our mixed xlm model achieves the best performance in all domains. in the monolingual setting experiment, mixed xlm also achieves comparable performance, which proves the robustness of the model and the effectiveness of brcc.