Language Identification and Normalization of Code Mixed E nglish and P unjabi Text (2020)

Abstract: Code mixing is prevalent when users use two or more languages while communicating. It becomes more complex when users prefer romanized text to Unicode typing. The automatic processing of social media data has become one of popular areas of interest. Especially since COVID period the involvement of youngsters has attained heights. Walking with the pace our intended software deals with Language Identification and Normalization of English and Punjabi code mixed text. The software designed follows a pipeline which includes data collection, pre-processing, language identification, handling Out of Vocabulary words, normalization and transliteration of English- Punjabi text. After applying five-fold cross validation on the corpus, the accuracy of 96.8% is achieved on a trained dataset of around 80025 tokens. After the prediction of the tags: the slangs, contractions in the user input are normalized to their standard form. In addition, the words with Punjabi as predicted tags are transliterated to Punjabi.

## introduction india is the second largest online market in the world, ranked after china with over 560 million internet users. 1 facebook is the largest social network with more than 2.7 billion monthly active users followed by whatsapp, twitter, and instagram. plenty of social media platforms are available nowadays but the most popular in context to indic languages are facebook, etc. @xcite was earliest to develop tools for automatic language identification by preparing a language learnability model. @xcite stated that code-switching occurs when a user switches between different languages in written or spoken a single instance. nowadays, code switching and code mixing are used alternatively. word level language identification is one of the challenging tasks as code mixing takes place at word level, at sentence level and even at sub word level in an utterance. challenges posed are numerous and keep changing with the intensity of languages in the utterance; still due to paucity of data the groundwork remains challenging. ## methodology the main focus of current research is to identify the language of every word in the english punjabi code mixed. the first and foremost task for developing the system is collection of code mixed social media text (english-punjabi) using api twitter threads for twitter, selecting some prolific users comments for facebook as data and some student community prolific users chat for whatsapp followed by cleaning of extracted data. @xcite used hindi, english, acronyms, universal tags along with code mixing index. @xcite used english, hindi and rest tags. in addition to the language tags @xcite discussed named entity and ambiguous tags. @xcite experimented with different possible combinations of available words, context and part of speech (pos) tags. @xcite have used hindi, english, universal, named entity, acronym, mixed and undefined tags. the dataset used in the current research consists of 80025 tokens (after preprocessing) which have been tagged as en (english), pb (punjabi), univ (universal), mixed (mixing of two languages inside a word), ne (named entity), acro (acronyms), rest (none of earlier mentioned tags). a supervised model is trained with conditional random fields (crf) which calculates the conditional probability of output tags given the values assigned to the input nodes. the features used are contextual features, capitalization features, special character features, character n-gram features and lexicon features. after applying five-fold cross validation on the corpus, the accuracy of 96.8% is achieved on a trained dataset of around 80025 tokens. in social media text people use creativity in spellings rather than traditional words. the deviation of text can be categorized as acronyms, slangs, misspellings, use of phonetic spellings etc. contractions like hasn't-has not, ma'am-madam etc. which are handled by mapping. plenty of common english words e.g. lyk -like, feb-february, gm-gud morning have changed their existence on social media. a dictionary of such out of vocabulary has been maintained in order to normalize them. a transliterated dictionary for the code mixed data contains transliterated pairs of romanized text and its punjabi equivalent. e.g kithey-ਕਿਥੇ , janam -ਜਨਮ etc. after the prediction of the tags: the slangs, contractions in the user input are normalized to their standard form words with punjabi as predicted tags are transliterated to punjabi language. ## results on the bilingual english-punjabi data set the crf baseline approach reports an accuracy of 97.24 % with f1-score 96.8 % on the english-punjabi language pair.