Corpus Creation and Language Identification in Low-Resource Code-Mixed T elugu- E nglish Text (2021)

Abstract: Code-Mixing (CM) is a common phenomenon in multilingual societies. CM plays a significant role in technology and medical fields where terminologies in the native language are not available or known. Language Identification (LID) of the CM data will help solve NLP tasks such as Spell Checking, Named Entity Recognition, Part-Of-Speech tagging, and Semantic Parsing. In the current era of machine learning, a common problem to the above-mentioned tasks is the availability of Learning data to train models. In this paper, we introduce two Telugu-English CM manually annotated datasets (Twitter dataset and Blog dataset). The Twitter dataset contains more romanization variability and misspelled words than the blog dataset. We compare across various classification models and perform extensive bench-marking using both Classical and Deep Learning Models for LID compared to existing models. We propose two architectures for language classification (Telugu and English) in CM data: (1) Word Level Classification (2) Sentence Level word-by-word Classification and compare these approaches presenting two strong baselines for LID on these datasets.

## introduction language is one of the significant aspects which makes humans different from other species. it is not a fixed entity, and it has evolved with time and will continue to do so. as a part of such an evolutionary process, we are at the stage of code-mixing where people communicate by mixing linguistic units such as phrases, words, and morphemes of one language embedded within an utterance of another language @xcite , @xcite . india has 22 officially recognized languages foot_0 and many dialects. for a land with such linguistic diversity, bilingualism and multilingualism is a prevalent trait. telugu belongs to the family of dravidian languages. it is primarily spoken in southern india and is also the third most spoken language in india. telugu is the mother language of a large population native of andhra pradesh and telangana states. english is a primary mode of teaching for most of the population across the globe. with such an influence of a language, people tend to use a mix of both english and their native language in an informal conversation (both speech and text). cm is classified into two types: intra-sentential and inter-sentential code-mixing @xcite . intra-sentential code mixing refers to the use of multiple languages in a single sentence. inter-sentential code-mixing is when the language switching is done at the end of the sentence. in this paper, we will focus on intra-sentential code mixing. in a multilingual setting, most of the conversations that happen informally are cm. cm is also used extensively in social media platforms, blogs, and forums as posts, chats, and comments. the processing of cm text poses an exciting and challenging problem to the linguistic community. this is because of the added complexities in the traditional processing tasks such as spell checking, named entity recognition (ner), parts-of-speech (pos), natural language generation (nlg) and machine translation (mt) due to the unavailability of prior information about the language at any point of time. intra-sentential lid is the task of identifying the languages of each word. researchers have made significant progress in the lid module in the automated processing of cm text. however, in low-resourced agglutinative languages like telugu, lid is still a challenging task @xcite . in this paper, we introduce two datasets and propose various pipelines to tackle intra-sentential language identification problem in cm data using deep learning models. this example sentence illustrates the cm addressed in this paper: example: elen/ne everyday/en school/en ki/te buslo/te velthundi/te ./univ (translation: elen goes to school by bus everyday) the words followed by /ne, /te, /en, and /univ correspond to named entity, telugu, english, and universal tags. in the above example, some words exhibit morpheme level code-mixing, like in "buslo" : "bus" (english word) + "lo" (plural morpheme in telugu). we also consider the clitiques like "supere": "super" (english root word) + "e" (clitique) as code mixed. the key contributions of this paper are the following: the rest of this paper is organised into 6 sections. in section 2, we discuss the related work. section 3 elaborates on challenges while working with cm data, followed by the dataset and its annotation in section 4. section 5 describes the approaches for lid in cm data. section 6 reports the results of the proposed ## related work a significant amount of work has been done recently in the field of code-mixed data, especially in the area of lid. @xcite discussed the syntax and structure of multilingual language organization and the role of language dependence in linguistic convergence of cm from an indian perspective. to create a word-level language identifier, king and abney (2013) used weakly semisupervised methods. according to noor al-qaysi (2017), code-switching is very popular on social networking sites such as facebook, twitter, and whatsapp and 86.40 percent of students use code-switching on social networks, while 81 percent of educators do so. yogarshi vyas and choudhury (2014) used logistic regression and a module that calculates code-switching probability. @xcite merged two classifiers into an ensemble model for hindi-english cm lid using multiple features such as word sense, dictionary, n-gram, and edit distance. the first classifier takes changed edit distance, word frequency, and character n-grams as features. the first classifier's output and the pos tag of the neighboring words are given as features to the second classifier to predict the final label. @xcite used the shallow parsing pipeline to perform successful text analysis in hindi-english cm social media data. for the task of lid, most of the experiments depend on dictionaries, supervised classification models, and markov models. the basic distinguishing features such as specific character combinations, repeated or unique words, diacritics, or typical n-grams are used in the simplest lid methods @xcite @xcite . some lid methods model sequences of words, characters, or bytes as model complexity increases. some approaches concentrate on modeling the frequency of n-grams, such as character n-gram frequency @xcite . these methods outper-form techniques that rely on one-of-a-kind terms. kalita and saharia (2018); veena et al. (2018) used support vector machines(svm) with linear kernel to perform lid. @xcite equipped crf based approach for lid on cm hindi-english and spanish-english text. in @xcite , efforts have been made to propose an accurate model to address the problem of classification in cm data. however, the recent advancement in deep learning models have led to better performance in various nlp tasks. we thus leverage these deep models for classification involving low resourced language (telugu) in cm data with the help of larger corpus which have been sourced from twitter and other popular blogs. ## dataset we created two different types of code-mixed datasets sourced from twitter foot_2 and popular blogs 45 using code-mixed language. as shown in for the twitter dataset, we manually identified 40 user accounts who tweet with cm; these user accounts often tweet on different aspects such as movies, politics, and sports. we used the twitter api to get 1000 tweets of each user. for the blog dataset, we scrapped chaibisket.com and wirally.com websites which provide high-quality code-mixed content. we manually picked 350 articles containing code-mixed data for this dataset. the pre-processing of the data involved the following steps: after preprocessing the data, it is then manually annotated into four classes, i.e., telugu (te), english (en), named entities (ne), and universal (univ). named entities include names of people, organizations, and locations. universal includes punctuation marks, acronyms, emojis, hashtags, and numbers. ## methodology in this work, we consider lid in cm data as a classification problem where we label each word to its corresponding language class. with the advent of deep learning, many popu-lar tasks like named-entity recognition, partsof-speech identification have shown promising results @xcite , @xcite . in section 5.2 we explain an extensive set of deep learning models for lid in cm data. we show the quantitative results of lid in section 6. we provide the following architectures based on the type of input the model is supplied with to solve this problem: the example given below illustrates the input and output from this approach. input: bagundi (translation: good) output: te ## results we have put forward an extensive set of deep learning models to tackle lid in english- ## conclusion and future work in this paper, we have put forward two telugu-english cm manually annotated datasets which are an order of magnitude greater than the existing dataset, and proposed two architectures for language classification (telugu and english) in cm data: (1) word level classification (2) sentence level word-by-word classification. we have conducted thorough experimentation and extensive benchmarking across various deep learning models and classical machine learning models. we found out that bilstm + lstm, bilstm + crf performs the best among others. we also plan to make our data corpus consisting of lowresourced telugu and english languages generated from twitter, and online blogs opensourced to encourage further experimentation and research. the lid models developed here can also be used in other nlp tasks like named entity recognition (ner), parts of speech (pos) tagging, and spell-check. the results of the current work are encouraging, and future work will be focused on using sequence level labeling sota models like attention in lid. we are also focused on developing spell-checking models which normalize the misspelled and romanization variability as a part of our future work.