Word Level Language Identification in Code-mixed K annada- E nglish Texts using traditional machine learning algorithms (2022)

Abstract: Language Identification at the Word Level in Kannada-English Texts. This paper de- scribes the system paper of CoLI-Kanglish 2022 shared task. The goal of this task is to identify the different languages used in CoLI- Kanglish 2022. This dataset is distributed into different categories including Kannada, En- glish, Mixed-Language, Location, Name, and Others. This Code-Mix was compiled by CoLI- Kanglish 2022 organizers from posts on social media. We use two classification techniques, KNN and SVM, and achieve an F1-score of 0.58 and place third out of nine competitors.

## introduction nowadays, it is impossible to find somebody who doesn't use social media or smartphones. this prompts us to identify a new difficulty for people who have used social media. the following difficulties are just one example of the many tasks are performed in natural language processing (nlp) to address various issues for instance, fake news @xcite , machine translation detection, sentiment analysis and language identification @xcite . identification language for mixed languages is the major challenge. many users want an easy way to construct sentences, or employ habitual expressions. they try to write in a combination of two or three different languages, which leads the creation of code-mix data @xcite . user-generated content like web articles, tweets, and message boards frequently contain code-mix text, yet majority of the language id models in use today have been ignored. as observed in these english-hindi examples, code-mixing entails language changes inside and across constituents. as many as 17% of facebook posts from india are code-mixed @xcite and 3.5% from tweets @xcite . nearly all social media networks where people speak several languages are code-mix. for instance, in a nation like india, where there are more than a dozen different languages with various alphabets, you may easily locate a mix-code of english and indian languages if you check the posts on facebook or twitter that are linked to garments or related to shopping @xcite . because of their extensive range, code mixes cannot be adequately described in a finite number of words. code-mixing may contain a variety of words including words that combine the alphabets of two languages that identify an area, a person or a place, and different situations. we will now discuss the classification system we utilized in this paper and also tf-idf vectorizer.one of the effective supervised machine learning techniques that we may utilize for both regression and data classification is called the support vector machine (svm). finding the hyperplane in an n-dimensional space that clearly classifies the data points is the objective of an svm. @xcite . this means that the decision boundry line between the data points that fall into a category and those that do not is drawn clearly by the algorithm. almost all data that is encoded as a vector is suitable for this technique. if it create a good vector from our data, we can use svm to find good results @xcite . although knn can be used just like svm for both classification and regression issues, it is the primary application in classification. this algorithm stores all the data and can classify a new data point based on similarities. this method places the new instance into the column that is more comparable to the available categories and makes the assumption that the new data is linked to the available items @xcite . as it encounters new data, this algorithm simply stores the data set during training and then classifies it into a group that is roughly similar to the present data. the tf-idf statistic gives keywords that can be used to identify or categorize particular documents by demonstrating the relevance of certain keywords to a given set of documents @xcite . ## task description and datasets language identification (li) is the process of automatically recognizing the languages used in a text. kannada is one of the dravidian languages that make up india's rich linguistic legacy and is used as the official language of the state of karnataka. karnataka residents can read, write, and speak kannada, yet many find it challenging to use the language while posting messages or comments on social media. language identification is the process of automatically recognizing the languages used in a given text because code-mixing is one of the most challenging subjects in natural language processing (nlp). the goal of the current investigation is to identify the language of the words. as part of this work, we must determine which words are of english, kannada, and mixed languages. the coli-kenglish dataset consists of kannada and english words written in roman script and is divided into six main categories: "kannada," "english," "mixed-language," "name," "location," and "other." participants are asked to submit their methods in the kanglish shared task, which requires that each word be recognized and categorized in one of these categories @xcite . ## related work language identification in social media texts is difficult because of things like social media content that has been code-mixed, using one alphabet to write in two languages at this point. @xcite proposed a code that combines dravidian data in kannada, malayalam, and tamil. @xcite extend a twitter data collection that include hinglish data. they provided primary experiment findings with an accuracy of 0.71 using classifiers support vector machine (svm) and random forest (rf) with n-grams and lexicon-based features @xcite . sentiment analysis for dravidian languages in code-mixed data was a shared task in dravidian-code-mix-fire2020 established by @xcite . researchers had submitted a variety of models, and they used the under sampling technique from @xcite to train some machine learning classifiers with various syntax-based n-gram features. the linear regression classifier with word and char n-gram features produced positive results with average weighted f1-scores of 0.71 and 0.62. ## methods in this study, we employed standard machine learning algorithms for language identification. for this task, we used two different classifiers, including (i) support vector machines and (ii) k-nearest neighbors. we also used n-gram tf-idf word and character features for vectorization. on each of these classifiers and this vectorization, we make a comment. for this task, we submit 5 runs, and the outcome varies each time. ## experiments and results we demonstrate our experiment with text data that was gathered from youtube. each language pair's word should be categorized into one of the six groups shown in ## conclusion this study shows how different languages may be identified in code-mix data using a classifier that uses two algorithms, knn and svm. the first technique produces better results, with the best weighted average f1-score 0.58.