B o NC : Bag of N-Characters Model for Word Level Language Identification (2022)

Abstract: This paper describes the model submitted by NLP_BFCAI team for Kanglish shared task held at ICON 2022. The proposed model used a very simple approach based on the word representation. Simple machine learning classification algorithms, Random Forests, Support Vector Machines, Stochastic Gradient Descent and Multi-Layer Perceptron have been imple- mented. Our submission, RF, securely ranked fifth among all other submissions.

## introduction language identification ("li") is the process of identifying the natural language that a document or a portion of it is written in @xcite . a human reader who is familiar with a language may easily recognize material written in that language. therefore, the goal of li research is to imitate the capacity of humans to identify these languages. the early attempts to solve this problem were made at the beginning of the century @xcite . after that, there are several computer methods is being used without the assistance of a human using specifically created algorithms and indexing structures. over the years, li research has developed methods to recognize human languages. li is applicable for all forms of information storage that incorporate language, whether digital or not, and applies to every modality of language, including voice, sign language, and handwritten text. however, we restrict the scope of this paper to li of written material that has been digitally encoded. the ability to identify the language for a written document has a wide range of uses such nlp tasks. it plays an important role in attracting users to a specific website that can provide relevant information for the user's native language @xcite . in information retrieval and storage, the procedure of indexing documents in a multilingual collection according to the language they were written in is common. li is required for document collections where the languages of the documents are unknown at the outset, such as for data retrieved from the world wide web @xcite . it is also suitable for machine translator applications by detecting the user's language without selecting it. language identification is useful also in security. forensic linguistics is one of the potential applications that use the li which is considered the link between the legal system and linguistic stylistics @xcite . li can be used as a methodology for authorship profiling to give proof of an author's linguistic background @xcite . authorities and intelligence agencies may be able to learn more about threats and their perpetrators if they can extract more information from an anonymous sms. investigators can identify the author of anonymous literature with the use of hints about their languages. in these situations, li can be used to discover the discriminant language cues in anonymous communication @xcite . the study of language acquisition and teaching has received a lot of linguistic attention. the need for resources for language learning has increased because of the growing number of language learners, which has in turn fueled most of the language acquisition research over the past ten years. the development of the second language acquisition (sla) theory may potentially benefit from the outcomes of an li task. the new detectionbased approach to transfer articulates the convergence of li approaches and transfer research @xcite , which was first proposed by @xcite . li can be used to create teaching strategies, guidelines, and learner feedback that are tailored to each student's mother language. the models specific to each language can be used to create this customized evaluation. for instance, algorithms based on these models could give students feedback in automated writing evaluation systems that is considerably more targeted and concentrated @xcite . a new word-representation model, bag of n-characters (bonc), has been presented in this work. the proposed model is a derivation of the bag of words model (bow) for characters. different machine learning algorithms namely; support vector machine (svm), stochastic gradient descent (sgd), random forest (rf) and multi-layer perceptron (mlp) have been implemented using bonc model as a vectorization technique. the rest of the paper is organized as follows: section 2 describes the dataset; section 3 describes the system architecture. experimental settings and results are given in section 4. finally, the conclusion and future work are presented in section 5. ## dataset the dataset, coli-kenglish, has been distributed by task organizers to the participants (hosahalli lakshmaiah et al., 2022). it comprises a set of english and kannada words written in roman script. the words are grouped into the following set of classes {kannada, english, m ixedlanguage, n ame, location, other} ## system architecture the general framework of proposed model consists of three main phases. the first phase is preprocessing where the raw words were prepared to further steps. the second phase is word representation and the third phase is model training. after model construction, the test set was fed to the model for model evaluation. the following are details of each phase. ## experiments and results for preprocessing phase, the threshold is set to be 4, @xmath0. the vector length was 64, i.e the character vocabulary contains 64 characters. kfolds cross validation technique has been used for the development phase. the training set is divided into three folds, at the first run the first fold has been used as test set and other folds as the training set and so on. ## conclusion and future work in this paper, a simple framework for language identification has been introduced. a vectoriza-tion approach (bonc) has been compared. it is clear from the results that rf outperforms all other classifiers. from this study, we can conclude that language identification of text is one of the challenging tasks. in future work, pre-trained models could be used to improve the performance of classification. transfer learning can be applied so that knowledge from one domain can be transferred to another domain. oren tsur and ari rappoport. 2007. using classifier features for studying the effect of native language on the choice of written second language words. in proceedings of the workshop on cognitive aspects of computational language acquisition, pages 9-16, prague, czech republic. association for computational linguistics.