BERT -based Language Identification in Code-Mix K annada- E nglish Text at the C o LI -Kanglish Shared Task@ ICON 2022 (2022)

Abstract: Language identification has recently gained research interest in code-mixed languages due to the extensive use of social media among people. People who speak multiple languages tend to use code-mixed languages when communicating with each other. It has become necessary to identify the languages in such code-mixed environment to detect hate speeches, fake news, misinformation or disinformation and for tasks such as sentiment analysis. In this work, we have proposed a BERT-based approach for language identification in the CoLI-Kanglish shared task at ICON 2022. Our approach achieved 86% weighted average F-1 score and a macro average F-1 score of 57% in the test set.

## introduction social media plays a big role in today's life. with the deep penetration of the internet among the masses, people use social media in all directions. in a region where people use different languages, mixing words or sentences from more than one language is very common. this also happens on social media where people exchange their views using code-mixed languages, most of the time in a common script like roman. @xcite defined code-mixing as the blending of words or sentences between two distinct languages within a single speech occurrence. it has emerged as a separate language phenomenon in a multilingual culture as a result of the increased usage of social media @xcite . although the problem of language identification is very old, major research has been done around the world on identifying languages in codemixed environments @xcite @xcite @xcite @xcite . however, in a code-mixed scenario, there are rela-tively few studies that have attempted to find regional languages from india. in this paper, we have explored the use of state-of-the-art nlp and deep learning techniques to identify language in the coli-kenglish dataset @xcite for the shared task coli-kanglish @xcite . we also share our code used for the experiments on github foot_0 . as a result of recent developments in nlp, a large number of language models built on the transformer paradigm have emerged @xcite . in terms of several nlp tasks, such as text categorization, natural language inference, question answering, and textual similarity, one such model, called bert, has produced state-of-the-art results @xcite . these models can be used for a variety of downstream tasks because they were trained on massive amounts of text data from sources like wikipedia and bookcorpus. for our work, we have used bert @xcite and deep neural networks for the kannada-english language identification task. our results evaluated on the test set show that using bert can produce good results, which shows the potential of such models for future related work. ## related work this section contains a brief discussion of some recent works on identifying languages in code-mixed language pairings for indian languages. similarly, @xcite proposed two different models cooli-ensemble and cooli-keras to identify and classify code-mixed texts of three language pairs, namely, kannada-english, malayalam-english and tamil-english into six predefined categories (5 categories in malayalam-english language pair). the proposed models have been trained with features extracted from sentences such as character sequences combined with words. the authors found that the cooli-ensemble model performed the best among the proposed models. another work by @xcite focused on malayalam-english code-mixed corpus at the word level using transfer models like camembert @xcite , xlm-roberta, electra @xcite and distilbert. the results of this study showed that electra performed better than the other models. another recent study on language identification for tamil code-mixed youtube comments was conducted by @xcite . the dataset was collected from youtube posts and comments in a multilingual environment. cnn-bilstm, distilbert and xlm-r models gave similar but poor results on this dataset, and ulm-fit attained a better performance over the other models due to its superior fine-tuning methods. they proposed a selective translation and transliteration for the code-mixed corpus. they also showed the advantage of using transformer based models on low resource languages. ## approach we first describe the specifics of the dataset that we use in this section. after that we will discuss the approach that we used using bert. we also compare the results among various bert-based models along with traditional machine learning approaches. ## experiment details for the bert experiment purposes, we have used different bert base models from huggingface foot_2 . we used the tensorflow foot_3 framework for our experiments. we report the results of our experiments on the annotated test set of the dataset. for defining our neural network model, we have used three dense layers on top of the bert embedding layer containing 128, 64 and 32 neurons, respectively, with relu activation function with a dropout rate of 0.2 at each layer. the final dense classification layer contains 6 neurons with a softmax activation function. the bert layer consists of the word embeddings from the bert-base model along with the input word ids and the masked sequence of the words. during the neural network model training we have used a learning rate of 2e-5 which is taken from the original bert paper @xcite . we used a maximum sequence length of 15, @xmath0, @xmath1.01 and a batch size of 128 is used for the training over 20 epochs. we keep the same experimental settings for all the models. for optimization, we have used the adam optimizer (kingma and ba, 2014) with a categorical cross entropy loss function ## results and discussion from the from the results we can see that using bert, identification of languages in a code mix kannada-english text corpus can be achieved with better results than traditional machine learning algorithms. since bert can learn word contexts, our objective for adopting it is validated. as a result, it performs better when it comes to detecting languages with more precision and recall. we have also compared our work with the top ranked teams for the coli-kanglish shared task. the results are shown in tables 5 and 6 . we can see that for the weighted average scores, our method has the same f-1 score as the top ranked team which is 86%. however, for the macro f-1 score, our method is lower than the rest of the teams with 57%. ## ablation study we also performed a few ablation studies where we dropped a few of the category tags. from the for our ablation studies, we first dropped only the "location" tag and performed the experiment with the bert-base-uncased model. we then dropped only the "name" tag and performed the same set of experiment. we then dropped both tags and performed the experiment. the results of these studies are shown in ## conclusion there is a large research potential for automatic language detection in code mix text. to spot hate speech or the dissemination of false information in a multilingual culture where speakers converse in a variety of languages, language identification is important. in this paper, we have used a bertbased approach to identify language in a kannada-english code mix corpus. we have seen improvements over traditional machine learning algorithms when using these models, paving the way for further research in this direction using such models. we have also seen that availability of more data can lead to increase in efficiency of such models.