Transformer-based Model for Word Level Language Identification in Code-mixed K annada- E nglish Texts (2022)

Abstract: Language Identification at the Word Level in Kannada-English Texts. This paper describes the system paper of CoLI-Kanglish 2022 shared task. The goal of this task is to identify the different languages used in CoLI-Kanglish 2022. This dataset is distributed into different categories including Kannada, English, Mixed-Language, Location, Name, and Others. This Code-Mix was compiled by CoLI-Kanglish 2022 organizers from posts on social media. We use two classification techniques, KNN and SVM, and achieve an F1-score of 0.58 and place third out of nine competitors.

## introduction in recent years, language identification of social media text has been a fascinating research topic @xcite . social media platforms have become more integrated in this digital era and have impacted various people's perceptions of networking and socializing @xcite . this influence allowed different users to communicate via various social media platforms using a mix of texts. nlp technology has advanced rapidly in many applications, including machine translation @xcite @xcite , abusive comment detection @xcite , fake news detection @xcite , aggressive incident detection @xcite , hope speech detection @xcite , and others. however, numerous tools have not yet been created for languages with limited resources or languages with code-mixed data. code-mixing is the use of linguistic units-words, phrases, and clauses-at the sentence or word level from various languages. in casual communication, such as social media, it is typically seen. we have access to an enormous amount of code-mixed data because of the various social media platforms that allow individuals to communicate @xcite . as a result, automatic language recognition at the word level has become an essential part of analyzing noisy content in social media. it would help with the automated analysis of content generated on social media. currently, in the area of nlp, different researchers are developing different nlp applications in code-mixed datasets. some of the applications are code-mixed sentiments analysis @xcite , code-mixed offensive language identification @xcite , etc. we took part in the kanglish shared task @xcite , which aims to identify language at the word level from code-mixed data for kannada-english texts. for word-level code-mixed language identification tasks, we used transformer -based @xcite pre-trained language models (plms). our transformer-based model consists of bert @xcite and its three variants. we used plms and lstm models for this word-level language identification task. this paper discusses a transformer-based model for word-level language identification in codemixed kannada-english texts for the kanglish shared task. the paper is organized as follows: section 2 describes past work related to this study, section 3 gives an overview of the dataset and its statistics, section 4 explains the methodology adopted in this study including the algorithms, section 5 emphasizes on the experimental results and descriptions. finally, section 6 concludes the paper. ## related work currently, solving nlp problems in code-mixed data is getting attention from many researchers. for word-level language identification in code-mixed text, different researchers have suggested various models. @xcite proposed a conditional random fields (crf)-based system for word level language identification of code-mixed text for four language pairs, namely, english-spanish (en-es), english-nepali (en-ne), english-mandarin (en-cn), and standard arabic-arabic (ar-ar) dialects. the authors explored various token levels and contextual features to build an optimal crf using the provided training data. the proposed system performed more or less consistently, with accuracy ranging from 80% to 95% across four language pairs. gundapu and mamidi (2020) also proposed a crf based model for word-level language identification in english-telugu code-mixed data. the authors used feature extraction as the main task for the proposed model. they used pos-tags, length of the word, prefix and suffix of focus word, numeric digit, special symbol, capital letter, and character n-grams (uni-, bi-, trigrams of words) as features. the proposed crf-based model had an f1-score of 0.91. a support vector machines (svm)-based model for word level language identification of tamil-english code-mixed text in social media is proposed by @xcite . the authors used dictionaries, double consonants, and term frequency to identify features. the proposed svm model with a linear kernel gave 89.46% accuracy for the language identification system for tamil-english code-mixed text at the word level. ansari et al. (2021) proposes transfer learning and fine-tuning bert models for language identification of hindi-english code-mixed tweets. the authors used data from hindi-english-urdu codemixed text for language pre-training and hindi-english code-mixed for subsequent word-level language classification. the authors first pre-trained hindi-english-urdu code-mixed text using bert and fine-tuned the trained model in downstream hindi-english code-mixed word-level language classification. their proposed model for hindi-english code-mixed language identification, both pre-training and fine-tuning with code-mixed text, gives the best f1-score of 0.84 as compared to their monolingual counterparts. table ?? shows some samples from the dataset used for training. word tag 0 anusthu kn 1 woww en 2 staying en 3 near en 4 hostel en 5 confirmed en 6 faith en 7 linked en 8 gtila kn 9 germany en ## experiments and results this section presents the description of the experimental setups, training parameters, results, and analysis. we conducted four experiments by replacing embedding layers with different pre-trained language models, the results are presented in section 5.2. ## conclusion in this paper, we explored the application of bertbased pre-trained language models to identify languages at the word level in code-mixed data for kannada-english texts. pre-trained models with a combination of the lstm model and a bertbased model outperformed the others and have shown promising results in identifying languages in code-mixed kannada-english texts. our team achieved the second place in coli-kanglish: wordlevel language identification in the code-mixed kannada-english texts competition. muhammad arif, atnafu lambebo tonja, iqra ameer, olga kolesnikova, alexander gelbukh, grigori figure 6: training and validation accuracy of bert based approach