hinglish N orm - A Corpus of H indi- E nglish Code Mixed Sentences for Text Normalization (2020)

Abstract: We present hinglishNorm - a human annotated corpus of Hindi-English code-mixed sentences for text normalization task. Each sentence in the corpus is aligned to its corresponding human annotated normalized form. To the best of our knowledge, there is no corpus of Hindi-English code-mixed sentences for text normalization task that is publicly available. Our work is the first attempt in this direction. The corpus contains 13494 segments annotated for text normalization. Further, we present baseline normalization results on this corpus. We obtain a Word Error Rate (WER) of 15.55, BiLingual Evaluation Understudy (BLEU) score of 71.2, and Metric for Evaluation of Translation with Explicit ORdering (METEOR) score of 0.50.

## introduction hindi is the fourth most-spoken first language in the world 1 . according to one estimate, nearly 0.615 billion people speak hindi as their first language 2 . of these people, most of the speakers are in india. the second most spoken language in india is english 3 . hindi and english are the official languages of the indian commonwealth 4 . a large number of these people have joined the internet recently. as a matter of fact, next billion users (nbu) is a term commonly used in tech and business circles to refer to the large number of people from india, brazil, china and south-east asia who joined the internet in the last decade 5 . this phenomena is primarily attributed to ubiquitous highly affordable phone and internet plans 6 . a large fraction of nbu users come from india and speak hindi as either their first or second language. a large number of these people use a blend of hindi and english in their daily informal communication. this hybrid language is also known as hinglish 7 . these users extensively use internet platforms which heavily rely on user generated content (ugc) -social media platforms such as facebook or twitter; messaging platforms such as whatsapp or facebook messenger; user reviews aggregators such as the google play store or amazon. a key characteristic of their behaviour on such platforms is their use of hinglish. thus, building any natural language processing (nlp) based internet applications for these users necessitates the ability to process this 'new' language. further, these ugc platforms are notoriously noisy. this means there is an additional challenge of non-canonical text. therefore, a key step in building applications for such text data is text normalization. intuitively, it is transforming text to a form where written text aligned to its normalized spoken form @xcite . more formally, it is the task of mapping non-canonical language, typical of speech transcription and computer-mediated communication, to standardized writing @xcite . separately, there has been a lot of work in the two areas of normalization and building corpora of hindi-english code mix text data, not much has been done at the intersection of the two(refer to section 2). to the best of our knowledge, there does not exist a corpus of hindi-english code mixed sentences for normalization where the normalizations are human annotated. this work is an effort to release such a corpus this work is motivated from our business use case where we are building a conversational system over whatsapp to screen candidates for blue-collar jobs. our candidate user base often comes from tier-2 and tier-3 cities of india. their responses to our conversational bot are mostly a code mix of hindi and english coupled with non-canonical text (ex: typos, non-standard syntactic constructions, spelling variations, phonetic substitutions, foreign language words in non-native script, grammatically incorrect text, colloquialisms, abbreviations, etc). the raw text our system gets is far from clean well formatted text and text normalization becomes a necessity to process it any further. the main contributions of this work are two-fold, viz. (i) creating a human annotated corpus for text normalization of hindi-english code mix sentences; and (ii) reporting baseline metrics on the corpus. further, we release the corpus and annotations under a creative commons attribution-noncommercial-sharealike license foot_6 . ## related work in this section, we present relevant work in the following areas viz.(1) text normalization (2) normalization and ugc datasets (3) code-mixed datasets, (4) hindi-english datasets. text normalization: text normalization, sometimes also called lexical normalization, is the task of translating/transforming a non-standard text to a standard format. using text normalization on noisy data, one can provide cleaner text data to downstream nlp tasks and improve the overall system performance @xcite , @xcite . some of the early work used a rule-based spell-checker approach to generate a list of corrections for any misspelled word, ranked by corresponding posterior probabilities (church and gale, 1991) @xcite . however, this approach did not factor in any context while normalizing words. @xcite used a hidden markov model (hmm), where they modeled each standard english word as a hmm and calculated the probability of observing the noisy token. "moses", a well known statistical machine translation (smt) tool, provided significant improvements in comparison to previous solutions @xcite . @xcite adapted a phrase-based machine translation (mt) model for normalizing sms and achieved significant gain in performance. in the past few years, neural network based approaches for text normalization have become increasingly popular and have shown competitive performance in shared tasks @xcite , @xcite . @xcite , @xcite and @xcite provide excellent literature covering the landscape on this topic. normalization and ugc datasets: @xcite introduced a text normalization approach for twitter data using a variety of supervised & unsupervised learning techniques. this study resulted in 'lexnorm' 9 , an open-source dataset containing 549 tweets. @xcite subsequently released lexnorm15 10 . this new dataset contained 2950/1967 annotated tweets in train/test sets. @xcite ) created the mtnt dataset foot_9 containing translations of reddit comments from the english language to french/ japanese and vice versa, containing 7k∼37k data points per language pair. this dataset contains user-generated text with different kinds of noise, e.g., typos, grammatical errors, emojis, spoken languages, etc. for two language pairs. (van der goot and van noord, 2017) introduced 'monoise', a general purpose model for normalizing ugc text data. this model utilizes aspell spell checker, an n-gram based language model and word embeddings trained on a few million tweets. it ## corpus preparation while preparing this corpus, we carry out the following steps. 1. data collection: collecting hindi-english sentences. ## benchmark baseline it is common to model the text normalization problem as a machine translation problem @xcite @xcite ) @xcite . given that bidirectional lstm with attention is a popular baseline model for machine translation task, we built a text normalization model using the same on the lines of work by @xcite . we evaluated our system using @xcite , bilingual evaluation understudy (bleu) @xcite and metric for evaluation of translation with explicit ordering (meteor) @xcite . ## availability the homepage for the dataset can be accessed here 23 . the new corpora we release are available for research and non-commercial use under a creative commons attribution-noncommercial-sharealike license 24 . ## conclusion & future work we presented hinglishnorm version 1.0, a corpus of hindi-english code mix sentences for text normalization task. thereby, filling a much needed gap. the purpose of this corpus is to serve as a benchmark dataset for evaluation of hindi-english code mixed text normalization model performance. we have also provided our benchmark baseline results on this corpus for comparison. as future work, we plan to build stronger baselines using sota models such as bert @xcite , distilbert @xcite , roberta @xcite , etc.