Normalization of Spelling Variations in Code-Mixed Data (2022)

Abstract: Code-mixed text infused with low resource language has always been a challenge for natural language understanding models. A significant problem while understanding such texts is the correlation between the syntactic and semantic arrangement of words. The phonemes of each character in a word dictates the spelling representation of a term in low resource language. However, there is no universal protocol or alphabet mapping for code-mixing. In this paper, we highlight the impact of spelling variations in code-mixed data for training natural language understanding models. We emphasize the impact of using phonetics to neutralize this variation in spelling across different usage of a word with the same semantics. The proposed approach is a computationally inexpensive technique and improves the performances of state-of-the-art models for three dialog system tasks viz. intent classification, slot-filling, and response generation. We propose a data pipeline for normalizing spelling variations irrespective of language.

## introduction there are around 6,500 languages spoken in the world today @xcite . english, mandarin, chinese tops the list with over 2 billion speakers around the globe hence are highly resourceful languages. on the other hand, there are resource-scarce languages such as polish, odia, hindi and, many more with few million speakers only. due to lack of resources understanding such languages poses a great challenge for the research community. natural language understanding (nlu) means extracting the semantic schema of the utterance to re-act according to the intent of the utterance. nlu is crucial for any human-tomachine interaction-based system such as chatbots, virtual assistants, and many more. now the mode of communication is restricted by the speaker's language and innate understanding of language systems. the goal-oriented dialogue system @xcite was first introduced based on dialog state tracking @xcite and gave a new direction to the nlu tasks. a number of datasets are available in diverse domains for like-wise downstream goal-oriented conversational data such as atis @xcite , snips @xcite , dstc @xcite , woz @xcite , etc. however, all of them are monolingual, i.e., available in the english language only. code-mixing is the form of linguistics where the conveyer uses two (or more than two) languages together so that some words of the low resource language replace the words from high resource language or vice versa. the usual trend is to mix english with any other regional language such as hindi (hindi + @xmath0, bengali (bengali + @xmath1, tamil (tamil + @xmath2, and many more. with the increase in multi-lingual speakers, code-mixing is very common in online platforms, social media, and day-to-day life @xcite @xcite . most common activities such as shopping, restaurant reservations, booking tickets, and so on all involve extensive use of code-mixing. for example, a hindi speaking @xcite . a significant proportion of the population worldwide is using code-mixed language over online platforms @xcite . the prime complication with linguistic diversity is such that there is no convention or protocol to refer to when it comes to code-mixing. hence, it depends on the writer's perception and knowledge of the phonics of the source language. @xcite grouped writing errors into two classes. first one is typographical that occurs when a character is substituted by the wrong character whose key is nearby in the keyboard or interchanging of the character order, for instance, merw paas(mere paas), kimd(kind), kys krna ha(kya krna hai), and more variations due to different reasons. other class is of cognitive errors that occur when the writer is unaware of the native spelling and semantics of that word. in this case, the wrongly spelled word is phonetically close to the correct word @xcite . we assume that the chances of typographical errors are less since the writer intends to avoid making such errors. this paper mainly focuses on cognitive errors and covers a major portion of typographical errors as well. in the case of code-mixed data, there are no rules that can lead towards achieving the correct spelling because there is no correct spelling. we can assume the most commonly used representation as correct and normalized text to get some contextual meaning. the introduction of external knowledge can also help to improve the results of spelling correction. we propose a computationally inexpensive novel technique to normalize spelling variations irrespective of the language of the bilingual speaker. ## related work divergence from the traditional spelling and having variation for the same word often carry some meaning @xcite . in computational linguistics, while dealing with digital forms of regional text forms, it becomes helpful to map all spelling variations (semantically identical) to the same point in embedding space. @xcite highlighted a detailed analysis on the same. they analyzed that the skip-gram model, which does not consider spelling variations, encode spelling variation patterns to some extent. also, the use of cosine similarities helped find a link between intentional variations and distance from the conventionally followed standards. historical writings face the identical problem of high degree variance in spellings @xcite -every day with new findings in historical text and extending the corpora in digital form (literature). various researches already explored the normalization approaches based on string distance measures to a reasonable extent for proposing various tools for normalization. @xcite shows that, individually, the rule-based method (norma tool) performed best in the presence of a large amount of training data @xcite . a combination of normalization methods produces the best results and helps in further cleaning and processing of data. hence, integrating simple word-to-word mappers always increase the overall performance. methods like edit distance or levenshtein distance @xcite ) needs a massive corpus of universally correct data. @xcite further gave improvisation for this problem with the use of bi-lstm network @xcite applied on a character level. multi-task learning with additional normalization (integration of mappers) improves the model's performance. their model outperformed the crf-based models and norma tool given in @xcite . extending the work keeping in mind the idea of integrating, @xcite proposed three approaches based on statistical, neural, and character-level machine translation to train the model concerning modern spelling variation standards. their model covered a holistic view of the word-to-word mappers. additionally, they proposed a simplistic approach of a statistical dictionary, similar to a word-to-word mapper in which they used the changing frequency of spelling on the training corpora. they also stated that the statistical machine translation approach gave better results than neural machine translation on small corpora. @xcite explored another low resource language which was (thai). in their work, they proposed a two-staged pipeline with neural contextual attention. using neural error corrector and seq2seq error corrector alleviates the problem of overcorrection. the phonetically motivated approach has also been explored a little by @xcite where they prioritized the phonetic key of the misspelled word over supplementary ones. a survey conducted for @xcite reports an excellent survey on joint intent classification and slot filling techniques. they nlu models of over a decade and gave a detailed comparison with the pros and cons of various techniques. concluding the stateof-the-art research, they provide multiple comparisons that best summarise the past work done along different dimensions, including the features, base approaches, and dataset domain used. hybrid phonetic neural models @xcite and bert @xcite models have also been explored to capture character-phonetic but they don't capture the code-mixed data. @xcite conducted a survey of spelling correction techniques. they studied the interactive process of error production and correction. all the major research assumes that the correct spelling of the miss-spelled word is native to the written language. in the case of code-mixed data, no such thesaurus exists. additionally, there is the absence of any particular set of rules that one can use for code-mixing, and it is complex to come up with such a method or protocol to translate one language into a romanized language. recently, @xcite came up with a method for sub-word level representation learning that is supplemented by the word level lexical variations in code-mixed languages. they evaluated the proposed architecture on a mix of european and indic languages (spanish, hindi, bengali, tamil, telugu, and malayalam). they proposed a hierarchically attentive transformer (hit) framework, a novel architecture to encode text semantic and syntactic features in an embedding space with efficacy. it learns word representations at the sub-word level using a fused attention mechanism (fame). it incorporates two major attention components. an outer product attention (opa) @xcite to extract higher-order character-level similarities and multi-headed self attention (msa) @xcite , a standard transformer module that computes a scaled query-key vector pair dot product. fame extends the msa module by including osa and calculates their weighted sum. the proposed model tries to embed semantically and phonetically similar words of a code-mixed language by capturing relevant information at a more granular level but lacks overall coverage of spelling variations. hit is computationally expensive with over trainable parameters 2.7m for sequence classification and over 1.4m for pos tagging. it misses the essence of layman language in text utterances. ## dataset used we will use a code-mixed version of the dstc2 dataset @xcite @xcite . they incorporated code-mixing in four regional languages hindi, bengali, gujarati, and tamil, romanized as english @xcite by crowd-sourcing the data for language translation from native speakers of respective languages. the data contains 50k utterances on the restaurant reservation domain, including getting reservations done or asking for information such as restaurant address, phone, etc. final data contains bot-to-human dialog conversations. the authors converted the raw data from audio format to text. for this task, the authors used automatic speech recognition (asr) modules. sentence cheap restaurant south west mein slots b-price o b-area i-area o intent inform representation: there are 3 slots, 5 possible areas, 91 cuisines, and 3 price ranges. workers acting as customers were requested to deviate the conversation in the middle of the dialog to various slots and their possible values to make data robust, less intuitive, and unconditional and avoid unnecessary patterns in data. the workers transcribed the conversations and labeled the utterances with different dialog states. for example, each utterance was labeled with its semantic intent representation (request[area], inform[@xmath3, requested slots (@xmath4, address) and the method of search (by constraints, by alternatives). such annotations are useful for domain-specific slot-filling based dialog systems @xcite . this whole process consists of three phases. extracting unique utterances from dstc2, i.e., dialogues with only change in the slot values rest same are filtered out. creating code mixed translations using amazon mechanical turk (amt) tool for crowd-sourcing and by in-house workers. the evaluation involved taking random dialogues from the dataset for colloquialism (unforced translation), intelligibility (easily understandable), and coherence(irrespective of neighboring utterances knowledge). the quantitative measure of code-mixing : as per the evaluation process. the authors analyzed the obtained code-mixed data for code-mixing measures. @xcite gave a metric to measure code-mixing in a sentence given as: here, the measure of code-mixing for sentence x, c u (x) is given by n (x), number do foreign language tokens in sentence. maximum number of tokens t in language l i from the set of languages l. in addition, number of language switch points given by p (x). in the above equation, the language of the majority of words in the sentence serve as embedding however, irrespective of the majority, we need to consider english as embedding and hindi as matrix language. to over come this the authors proposed to replace the general max here, t l i is the number of tokens in native language(hindi). also, the term δ(x i ) with values 0(if switch in matrix language) or 1(purely english) to measure the extent of inter-utterance code-mixing and frequency. the authors also considered the fraction of code-mixed utterances s:cm u tterances u :t otalu tterances . and the final equation is given as c u (x) for one utterance and c c (x) for complete corpus: ## proposed methodology we propose a novel method of robust systems for low resource similar token spelling variations. the main reason for little spelling variations in the pronunciation and ambiguous phonetics of the word. due lack of any formal conversion criterion, the writer introduces variations. the aim is to map the semantically similar romanized words such as kabhi, kabi, kbhi, etc together so that they can be neutralized to a single term and have identical embedding. then give a most common and close to figure 3: utterance count vs hindi word count ## experiments we perform numerous trials and runs to prove the importance of spelling normalization. we took a downstream task of intent classification and slot prediction. we use the bio-tagging format to transform the data for performing joint learning. we then compare the performance of state-of-theart algorithms on our normalized data and old data with variations. ## implementation we report bleu-4 @xcite and rouge-1, rouge-2 and rouge-l scores @xcite for natural language generation machine translation task to compare the results with the dataset baselines given by @xcite . further, we compute precision, recall, and f1 scores for intent classification and slot filling evaluation. we calculate weighted and macro average scores and then chose macro overweighted because this is a class imbalance problem. the weighted average will give significant weightage to the most frequent class whose performance may lead to 100%. to encounter this issue, we report macro averaged scores, i.e., an average of independent scores for each class, treating all classes equally. further, we play with different combinations of the state-of-the-art algorithms and compared the performance on both syntactically normalized (spell) and un-normalized inc spell versions of the dataset. we represent the data as each line containing a token and utterances separated by an empty line. each token corresponds to a bio tagging label, where b refers to the label's beginning, i refers to the intermediate words of the label, and o refers to other classes (no label of interest). ## results and comparison the khapra, 2019) both results in higher performance increased by 0.5 -1.5 units, when normalised data is used. normalization forces the vector representations for semantically identical but syntactically close terms to overlap each other in multidimensional embedding space and forces the model to treat all possible variations as the same only. ## conclusion with this emerging trend of code-mixing over social media platforms and daily communication in almost every region, the necessity to develop efficient natural language understanding models has increased. we communicate in a specific language (say, english) because the language has set standards, semantics, syntactic, and phonetics. without any standard for low-resource languages, it is tough to communicate. different people have different ways of pronouncing and depicting the alphabet, phonetics, and accent of a language close to their mother tongue. this action may lead to spelling variations while writing text as a medium for communication. we introduce a novel, computationally inexpensive, fully robust, and efficient method to normalize these spelling variations that work as an auto-correct to counter this problem. this mechanism helps in performance gain not only machine translation but also generic natural language tasks and any downstream task involving code-mixing problems. we give a universal mapping for hi-english code-mixing that can be used directly by making a query in o(1) time to normalize non-english words. it is evident from our work that integrating several modules together helps in performance improvement. we can further improve this solution by learning a model to further normalize out of the vocabulary terms. li yujian and liu bo. 2007. a normalized levenshtein distance metric. ieee transactions on pattern analysis and machine intelligence, 29(6):1091-1095. xiaodong zhang and houfeng wang. 2016. a joint model of intent determination and slot filling for spoken language understanding. in proceedings of the twenty-fifth international joint conference on artificial intelligence, ijcai'16, page 2993-2999. aaai press.