Language Identification at the Word Level in Code-Mixed Texts Using Character Sequence and Word Embedding (2022)

Abstract: People often switch languages in conversations or written communication in order to communicate thoughts on social media platforms. The languages in texts of this type, also known as code-mixed texts, can be mixed at the sentence, word, or even sub-word level. In this paper, we address the problem of identifying language at the word level in code-mixed texts using a sequence of characters and word embedding. We feed machine learning and deep neural networks with a range of character-based and word-based text features as input. The data for this experiment was created by combining YouTube video comments from code-mixed Kannada and English (Kn-En) texts. The texts were pre-processed, split into words, and categorized as ‘Kannada’, ‘English’, ‘Mixed-Language’, ‘Name’, ‘Location’, and ‘Other’. The proposed techniques were able to learn from these features and were able to effectively identify the language of the words in the dataset. The proposed CK-Keras model with pre-trained Word2Vec embedding was our best-performing system, as it outperformed other methods when evaluated by the F1 scores.

## introduction language identification (li), the process of automatically recognizing the language(s) present in a given document, is a key component of several text processing pipelines. the main task in li is text classification, which is the act of assigning text to categories represented by a finite number of labels. as social media facilitate rapid information exchange and generate large amounts of text, dealing with the many different languages in documents obtained from social media users is one of the challenges in natural language processing (nlp). code mixing is the practice of using multiple languages at once, changing the lexical and grammatical aspects of informal communication, especially on social media. social media users around the world often combine multiple languages to express their opinions online. nlp, a branch of artificial intelligence, plays an important role in understanding the language in which humans write and speak. in some circumstances, it is very difficult to identify languages in texts collected from social media, but computational methods can be applied to automatically recognize these languages. stateof-the-art methods @xcite @xcite in nlp tasks apply word embedding and n-gram-based models at the character or word level for different tasks, including li. there are several countries where different natural languages are spoken. language diversity has a great impact on people in marriage, sports, business, medicine, and education. in india, officially known as the republic of india, citizens have the ability to read, write, and speak various languages. the country is one of the largest by population, and citizens can communicate in a variety of languages including kannada and english. code mixing, which is the practice of switching between two or more languages, is widespread in multilingual societies like india. people frequently communicate in more than one language and not always in the language in which they are addressed. the multi-language texts of users in these multilingual societies are often difficult to understand and analyze. with several understudied low-resource languages, the challenge of li in code-mixed text is far from being solved. with this motivation, our task was to develop and evaluate models that can correctly classify labels in code-mixed kannada and english texts. machine learning algorithms (hosahalli lakshmaiah et al., 2022) @xcite @xcite , as well as deep learning algorithms @xcite @xcite , have been used in many sequence 1 1 labeling tasks in nlp. to identify language at the word level in kannada-english code-mixed text, we proposed two machine learning techniques submitted to the "coli-kanglish: word-level language identification in code-mixed kannada-english texts" shared task @xcite , namely ck-multiplex and ck-keras. the developed models were applied to the coli-kenglish dataset where the language of each word was detected and classified into a specified category. the next section highlights the background and previous research on code mixing in social media texts. the dataset used to investigate the code mixing between english and kannada is then introduced in section 3. the methods applied to recognize word-level languages are discussed in section 4. the language detection experimental results are shown in section 5, and the conclusions and future work are presented in section 6. ## background and related work a number of machine learning and neural networks have been used to tackle and improve various nlp tasks, including the classification of code-mixed languages. code mixing, according to @xcite , refers to a situation in which words and grammar from two or more distinct languages are combined in a single sentence. in addition, code mixing is used while speaking two different languages at the same time. it suggests that all lexical and grammatical components indicate the act of switching languages, and that code mixing is most common in informal settings and occurs when the conversants use both languages concurrently. @xcite ) offered a method that was applied to the facebook, twitter, and whatsapp dataset to identify the language of the text that has been mixed with hindi and english. certain sub-classes of the quantum lstm network model have been shown to be able to accurately learn and predict language in a text on social media. the obtained results pave the way for further use of machine learning methods in quantum dynamics without relying on the precise form of the hamiltonian. to identify the language of twitter data, @xcite conducted an extensive experiment using transfer learning and fine-tuning of bert models. for language pre-training and word-level language classification, the study uses a data set consisting of code-mixed texts in hindi, english, and urdu. the findings demonstrate that pre-trained representations on code-mixed data perform better than their monolingual counterparts. @xcite addresses the issue of mixedscript identification for a dataset that comprises roman urdu, hindi, saraiki, bengali and english. rnn and word vectorization were used to train the language identification model. furthermore, numerous model architectures were optimized, such as long short-term memory (lstm), bidirectional lstm, gated recurrent unit (gru), and bidirectional gated recurrent unit (bgru), and experimentation yielded a very good performance score. the study also looked at multilingual challenges including roman words fused with english letters, generative spellings, and phonetic typing. for a code-mixed text in english-bodo-assamese, @xcite was able to identify the language of the text at the word level. several classification methods were applied to analyze and predict the language of text collected from facebook. the n-gram and dictionary-based features were used to train the models on the codemixed corpus and yielded different accuracies for the word-level language detection task. for word-level language detection in code-mixed text, @xcite developed a crfbased system. their method can be replicated on different languages since it takes advantage of lexical, contextual, character n-gram, and special character features. the experimental results show that the crf-based technique performs consistently across language pairs when its performance is compared to other datasets. to identify language boundaries at the word level, @xcite conducted a study using chat message datasets in mixed english-bengali and english-hindi languages. the author introduced a code-mixing index to evaluate the level of mixing in the corpora and evaluated the performance of the system to multiple languages. @xcite proposed several techniques to learn the sequence of characters that are frequently swapped for others in standard transliterations. the authors demonstrated how these algorithms can do better than others in identifying hindi words that correlate with the transliterated words supplied. their distinctive experimental model for word-level language identification considers the language and part of speech of nearby words. the experimental findings indicate that the proposed 2 2 model performs better in terms of accuracy than the previous methods. to help machine learning (ml) classifiers tackle the issue of offensive language identification (oli) in code-mixed and multi-script texts, @xcite proposed the use of relevant features of syllables and character n-grams. three pairs of dravidian languages, malayalam-english, tamil-english, and kannada-english, were used to evaluate the performance of the proposed models. syllable and character n-gram features performed well for codemixed and multi-script text analysis, as shown by the results of ml classifiers. @xcite developed a unique architecture for code-mixed data language tagging that uses multichannel neural networks that mix cnn and lstm for code-mixed data word level language identification. this architecture incorporates context information. the multichannel neural network performed well in the language identification task when used with a bi-lstm-crf context capture module. ## dataset the words in the coli-kenglish dataset, provided by the shared task organizers, were written in kannada, english, or a combination of the two languages and are classified into six main groups: "kannada", "english", "mixed-language", "name", "location", and "other". the data was scraped from kannada youtube video comments and preprocessed according to @xcite . the unstructured texts with incomplete sentences and shortened words were codemixed between the two languages. two native kannada speakers carefully tagged 19,432 unique words extracted from more than 7,000 sentences to create the coli-kenglish dataset. ## methodology in different text classification tasks, numerous algorithms have been proposed and yielded promising results. the models predicted the categories of the words in the vocabulary based on the feature re-sponses received from the vector representation of the text. data cleansing, word segmentation, and tokenization are typically the pre-processing steps applied to the raw input text and used to train the models. the text representation transmits the preprocessed text in the form of @xcite , bag-of-words (bow) @xcite , term frequency-inverse document frequency (tf-idf) @xcite , and word2vec representations @xcite that the models can understand while minimizing information loss. word2vec model @xcite generates word vectors for semantic meanings using local contexts. a word vector is a fixed-length realvalue vector that is used to represent any word in the corpus. word2vec employs two critical models: cbow and skip-gram. the first way entails guessing the term that is being used on the assumption that its context is understood. when the word in use is known, the latter predicts the context. the word2vec training approach helps the system learn vector representations of words using the structure of the neural network. the proposed techniques implement systems based on well-researched methodologies such as character ngram and word2vec embedding. ## results after applying our language identification models to the dataset, we were able to classify the words in the test set according to the categories that the models had learned from. the languages were identified using random forest classifiers and multi-layer perceptron baseline models and the results are encouraging. we also used lstm's deep learning model to learn a better feature from the text. lstms were further trained using random initialized word embeddings. the systems were evaluated using accuracy, recall, and f1 scores, and the results obtained are shown in tables 3 and 4 . ## conclusion and future works in this study, a preliminary investigation was conducted to determine the language used in code mixing during interaction on social media. the vocabulary and grammar of code-mixed texts are often adapted from multiple languages, and new structures are frequently developed based on the language and usage habits of its users. we tackle the problem of language identification at the word level in code-mixed social media text containing english and kannada languages. we use a two-step classification approach for the word-level language identification task. the embedding of character, sub-word, and word-level information can assist in the learning of meaningful correlations in words from many different languages. the lstm recurrent neural network and word2vec embedding approach achieved the highest f1 score among the proposed models. in the future, we plan to use more deep learning models and text from other languages. in order to extract useful information from codemixed texts and make code-switching systems better understand reviews, comments, inquiries, sentiments, etc., it is necessary to adequately detect and process the language of these texts.