Overview of C o LI -Kanglish: Word Level Language Identification in Code-mixed K annada- E nglish Texts at ICON 2022 (2022)

Abstract: The task of Language Identification (LI) in text processing refers to automatically identifying the languages used in a text document. LI task is usually been studied at the document level and often in high-resource languages while giving less importance to low-resource languages. However, with the recent advance- ment in technologies, in a multilingual country like India, many low-resource language users post their comments using English and one or more language(s) in the form of code-mixed texts. Combination of Kannada and English is one such code-mixed text of mixing Kannada and English languages at various levels. To address the word level LI in code-mixed text, in CoLI-Kanglish shared task, we have focused on open-sourcing a Kannada-English code-mixed dataset for word level LI of Kannada, English and mixed-language words written in Roman script. The task includes classifying each word in the given text into one of six predefined categories, namely: Kannada (kn), English (en), Kannada-English (kn-en), Name (name), Lo-cation (location), and Other (other). Among the models submitted by all the participants, the best performing model obtained averaged-weighted and averaged-macro F1 scores of 0.86 and 0.62 respectively.

## introduction south asia is the most linguistically diverse region in the world that embodies more than 650 different languages 1 and india is a multilingual country having a rich heritage of languages in south asia. kannada is one of the dravidian foot_1 languages as well as scheduled languages of india and the official and administrative language of karnataka state with more than 40 million native kannada speakers. a significant number of people in this region are comfortable using english in addition to their native/local/regional language for the day-today communication. these multilingual speakers preferably use multiple scripts and/or languages to post their comments/ideas/opinions on social media platforms, making code-mixing a default language on social media. code-mixing can be carried out at the paragraph, sentence or word level and even at subword level @xcite . people usually mix their native and/or local language with english and prefer to write the content mostly in roman script rather than using the native script as most of the keyboard layouts of computers and keypads of smartphones have roman alphabets by default @xcite . people who write kannada find difficult to use kannada script while posting comments/reviews on social media mainly because of the difficulty in keying the consonant conjuncts (ottaksharas) and consonants with the secondary forms of vowels (gunitaskaras) @xcite , using roman keyboards/keypads and hence prefer to use roman script on social media @xcite . the situation remains the same for most of the indian languages as they have their own script. social media platforms have given their users the freedom of writing text very casually without following the grammar or syntax of any language. this has resulted in a huge volume of user-generated content which includes incomplete words and/or sentences, catchy phrases, userdefined short forms for words (e.g., 'gn8' for 'good night'), different slangs (e.g., meme, gmeet), abbreviations ('omg' for 'oh my god'), recurrent characters ('soooooo sad' for 'so sad'), etc. the presence of these informal words in any text makes it difficult to understand the content @xcite . further, a code-mixed scenario where words of one language are transcribed with words of other languages as prefixes or suffixes creates a lot of problems to analyze the text, particularly due to conflicting phonetics. the increasing number of social media users is increasing the user-generated content which makes it difficult to handle this text manually @xcite . this demands the tools and techniques that can process the user-generated text automatically for various applications. the preliminary step in handling code-mixed text for many of the natural language processing (nlp) tasks like machine translation (patel and parikh, 2020), parts-of-speech tagging @xcite , sentiment analysis @xcite @xcite , emotion analysis @xcite , hate speech and offensive language identification @xcite , hope speech detection @xcite , identification of native language @xcite , etc., is identifying the language of each word/phrase/sentence. several research works in li tasks have been carried out focusing on high-resource languages like french-english, spanish-english, and german-english. however, very little attention is given to the low-resource indian languages. furthermore, code-mixing is quite common in a multilingual country like india where many people are bilingual and english is considered as the official language along with the local/administrative language. hence, in india, code-mixing is mostly observed between any indian language and english in social media text @xcite . the rapidly increasing code-mixed content on social media in indian languages in general and kannada-english in particular requires efficient methods to perform li at word level. ## literature review recent decades have witnessed the immense interest of researchers in code-mixed text specifically for low-resource and under-resource languages and few li works have also been carried out as a part of handling such code-mixed text. word level li is modeled as a typical supervised learning problem and various machine learning (ml) and deep learning (dl) algorithms are experimented for the same. some of the relevant works are described below: @xcite developed learning models for word level li of hindi-english code-mixed data using feature vectors generated by the continuous bag of words (cbow) and skipgram models. they experimented with various ml models including: support vector machine (svm), random forest (rf), logistic regression (lr), gaussian naive bayes (gnb), k-nearest neighbor (knn), and adaptive boosting (adaboost), on the dataset consisting of 7,210 words selected from the corpus prepared by @xcite . among all the models, svm classifiers obtained the highest accuracies of 67.33% and 67.34% using cbow and skipgram models respectively. a word level li in code-mixed telugu-english text proposed by @xcite , tokenized 1,987 telugu-english code-mixed sentences obtained from the international conference on natural language processing (icon) 2015 shared task dataset foot_2 and manually tagged the tokenized words with parts-of-speech (pos) and li tags. by using previous, current and next words and their pos tags, length of the word, and character n-grams in the range @xmath0, 3) as features, they trained conditional random field (crf) classifier to perform word level li and obtained an accuracy of 91.28%. @xcite proposed a multichannel neural network (nn) model for li of codemixed hindi-english and bengali-english text using contextual information. they selected 6,000 instances from the dataset developed by @xcite and @xcite for hindi-english and bengali-english respectively and implemented multichannel neural associations by combining convolutional neural network (cnn) and long short-term memory (lstm) models coupled with bilstm-crf for word level li. their proposed models obtained accuracies of 93.32% and 93.28% for hindi-english and bengali-english data respectively. @xcite created a dataset for word level li in codemixed english-malayalam text and implemented transformer-based models for li. the authors extracted 50k code-mixed english-malayalam comments from youtube and tokenized them to obtain 7,75,430 words. these words are then annotated with the language to which they belong to using an unsupervised approach. transformerbased multilingual bidirectional encoder representations from transformers (mbert), cross-lingual language model for robustly optimized bert (xlm-roberta), camembert, distilled version of bert (distilbert), and efficiently learning an encoder that classifies token replacements accurately (electra) models, are fine-tuned to perform li. among all the models, fine-tuned elec-tra model performed best with an f1 score of 0.9933. word and character embedding-based learning models presented by @xcite for li of code-mixed hindi-english text are experimented on icon 2016 shared task dataset @xcite ) that consists of 772, 1,096, and 763 sentences from facebook, twitter, and whatsapp respectively. by collecting additional code-mixed hindi-english text from other resources, word and character ngrams are used to train three skip-gram models with @xmath1, 3, 5 which in turn is used to train the svm models for li. compared to various svm models trained on word-based and character-based embeddings, svm classifier trained on characterbased 5-gram embeddings obtained better accuracy. even though few research works are carried out in low-resource indian languages like kannada, tamil, telugu, etc., no works have been reported on word level li in code-mixed kannada-english text. this provides scope for research at word level li in code-mixed kannada-english text. ## task description the task of automatically identifying languages used in a given text is called li and it is a preprocessing step for many applications. li at the word level can be viewed as a sequence labeling problem where each and every word in a sentence is tagged with one of the languages in the predefined set of languages. despite a lot of work being done in li, the problem of li in the code-mixed scenario is still a long way from being illuminated @xcite . to address word level li in code-mixed kannada-english texts, these texts are extracted from kannada video comments in youtube to construct coli-kenglish (shashirekha et al., 2022) dataset. ## dataset comments for kannada videos in youtube are scraped using the youtube-comment-downloader foot_3 and are used to build coli-kenglish dataset @xcite . the scraped texts contain around 1,00,000 comments from 373 kannada youtube videos. preprocessing involves the removal of duplicate comments and comments written only in kannada script. after preprocessing, the total number of comments amounts to 72,815. the nature of comments are generally in one of the following forms: the annotated coli-kenglish dataset contains 19,432 unique words extracted from nearly 7,000 sentences that are categorized into 6 classes, namely: 'kannada', 'english', 'mixed-language', 'name', 'location' and 'other'. while 'kannada' and 'english' classes represent kannada and english words respectively, 'mixed-language' class represents words created using a combination of kannada and english in any order. 'name' class represents the names of persons and 'location' class the names of locations or places. any other words are represented as an 'other' class. the words described by 'mixed-language' pose a real challenge to li task as these words are framed by various combinations of english/kannada words and kannada/english affixes (prefixes and suffices). the beauty as well as the complexity of these mixed-language words lies in the word pattern created by an individual posting comments on social media. description of the class labels and their samples along with the english translation are presented in ## evaluation metrics in the case of an imbalanced dataset, categories with a larger number of samples affect the averagedweighted scores and could be high always. therefore, reporting only weighted scores could provide misleading information about models' performance. hence, inspired by @xcite , code-mixed li models for imbalanced coli-kenglish dataset are evaluated using macroaveraged and weighted-averaged f1 scores. ## baselines coli-ngrams -the best performing model proposed by @xcite employ a feature engineering module that generates a feature set of prefixes and suffixes of length 1, 2 and 3 along with char n-grams (@xmath2, 3, 5) from words, and byte-pair encoding (bpe) embeddings of sub-word ngrams (@xmath3, 2, 3). the extracted features are vectorized using tfidfvectorizer foot_4 to train linear svm (lsvm), multi-layer perceptron (mlp), and logistic regression (lr) classifiers. these three models are used as baselines in this shared task. all the models are trained with default parameters. ## overview of the submitted approaches thirty different runs are submitted by eight different teams for the kanglish 2022 shared task and eventually six teams submitted their working notes. figure 1 refers to the different learning approaches used by the participants in this shared task to submit the runs. the findings indicate that, while 54% of the participants experimented different transformers, 27% used traditional ml models and the remaining used dl models. figure 2 shows that about 46% of run submissions are made by employing pretrained language model (lm) or pretrained embeddings and 27% did not use any pretrained models for the task. team tiya1012 presented a transformer-based model by fine-tuning distilbert-based-cased model on the coli-kenglish dataset and obtained 0.62 averaged-macro f1 score and ranked first in the competition. team abyssinia experimented different lm models, namely: bert, mbert, xlm-r and roberta from huggingface with a lstm architecture. among all the lm models, both mbert and xlm-r with an averaged-macro f1 score of 0.61 outperformed the rest of the models and also ranked second in the shared task. team pdnjk also explored several transformerbased models for the task of li in code-mixed kannada-english words and their best performing figure 2: pretrained models used by participants model using bert scored an averaged-macro f1 score of 0.57 and ranked fourth in the shared task. team habesha trained character-level lstm and bilstm models with attention that reads the text as a sequence of characters. the proposed bilstm model outperformed the lstm model and obtained an averaged-macro f1 score of 0.61 and ranked second in the shared task. team lidoma explored character n-grams to generate character tf-idf to train traditional ml classifiers. among all the classifiers they explored, the highest performance of an averaged-macro f1 score of 0.58 was reported with a simple knn classifier. similarly, bag-of-characters were turned into character vectors by team nlp_bfcai. they introduced a character representation called bag-of-n-characters model which has very similar structure to character n-grams and experimented several traditional ml algorithms. eventually, the rf model on the proposed features obtained the highest averaged-macro f1 of 0.43. ## results and discussion the best results obtained for each team among all the predictions submitted by them are presented in other findings indicate that, all teams who employed nn and transformer models outperformed the baselines and other traditional ml classifiers. in general, the higher weighted scores are the results of successful predictions for pure english and kannada words and the difficulty on identifying mixed-language words and less frequent entities resulted in less performance for macro scores. most of the teams relied on multilingual transformers or only character n-grams for solving the problem of li in code-mixed text. this reveals that the participants have only a shallow understanding of code-mixed texts. no method was used by the participants that could directly target the issue of codemixed texts except the multilingual transformers that partially handled the task. ## conclusion the task of li is a primary step for many nlp tasks that are usually overlooked for low-resource languages. however, the recent advancement in technologies caused a rapid increase in the volume of texts in low-resource languages. these texts on so-