Normalization and Back-Transliteration for Code-Switched Data (2021)

Abstract: Code-switching is an omnipresent phenomenon in multilingual communities all around the world but remains a challenge for NLP systems due to the lack of proper data and processing techniques. Hindi-English code-switched text on social media is often transliterated to the Roman script which prevents from utilizing monolingual resources available in the native Devanagari script. In this paper, we propose a method to normalize and back-transliterate code-switched Hindi-English text. In addition, we present a grapheme-to-phoneme (G2P) conversion technique for romanized Hindi data. We also release a dataset of script-corrected Hindi-English code-switched sentences labeled for the named entity recognition and part-of-speech tagging tasks to facilitate further research.

## introduction linguistic codeswitching (cs) is the phenomenon of mixing two or more languages in the context of a single utterance. multilingual speakers around the world engage in codeswitching on a regular basis. codeswitched data differs from monolin gual data to a great extent which discourages use of existing nlp technologies on codeswitched text. codeswitching also combines the syntax and lexicon of the languages used, making it difficult for monolingual models to adapt to codeswitched data @xcite . in textual codeswitching, text is frequently romanized 1 due to various technical constraints. this is especially true in the case of hindienglish since the devanagari script for hindi is not widely available or efficient on modern technology. fig ure 1 shows an example of a codeswitched hindi english tweet. as we can see in the example, key board layouts force users to choose a single script 1 throughout this paper, we use romanized to mean transliterated to the roman script ## background user generated codeswitched data is noisy and rid dled with word variations, spelling mistakes, and grammatical errors. since the latin script does not possess all the consonants and vowels required to transliterate hindi, users come up with the most convenient ways to transcribe hindi. common variations in transliterated hindi are: besides the above, there are other non standard variations observed in transliterated hindi as well. these variations make it difficult to properly transliterate text using simple phonetic mappings due to the lack of a standard transliteration scheme. numerous schemes like wx notation @xcite , brahminetitrans @xcite , and others have been introduced. however, none of these have been widely em ployed by the general public. ## methodology we follow a two step system to transliterate ro manized hindi to the devanagari orthography. first, we normalize the input using a sequenceto sequence model. then, for the backtransliteration task, we syllabify the token and transcribe to de vanagari. for the graphemetophoneme task, we directly map the normalized tokens into the inter national phonetic alphabet (ipa). ## data we use the hinglishnorm dataset by @xcite to train the normalization model. the dataset comprises of romanized codeswitched sen tences and their normalized forms annotated by hu mans. the data contains both hindi and english tokens along with their normalized forms. we cre ate pairs of tokens and their normalized forms to train our model. we further augment the dataset with some frequently encountered hindi words on social media and their variations. ## experiments 6.1 normalization rule based systems are not the most efficient solu tion to normalization since they are not capable of capturing all possible variations. instead, we treat normalization as a general machine translation problem. we train a character level sequenceto sequence model for normalization following the ar chitecture of @xcite . the model is comprised of a long shortterm memory(lstm) encoder and lstm decoder. we use the keras li brary @xcite for training the model. ta ble 1 compares our model's performance with the baselines provided by @xcite . we evaluate our system using word error rate @xcite , bleu score @xcite , and meteor score @xcite . model wer bleu meteor @xcite it is likely that some of the errors are due to inconsistencies in the transcription scheme in the hinglishnorm dataset since it is annotated by hu mans. one such instance is the long vowel आ which is normalized to "aa" through most of the data. however, in some instances, the annotators normalize it to "a". for example, "bt control to krna pdega" from the training data is normalized to "but control to karana padega". a sample nor malized output is shown in original bhai hin wher r u frmm translation brother, where are you from? normalized bhaai where are you from