Using Alternate Representations of Text for Natural Language Understanding (2020)

Abstract: One of the core components of voice assistants is the Natural Language Understanding (NLU) model. Its ability to accurately classify the user’s request (or “intent”) and recognize named entities in an utterance is pivotal to the success of these assistants. NLU models can be challenged in some languages by code-switching or morphological and orthographic variations. This work explores the possibility of improving the accuracy of NLU models for Indic languages via the use of alternate representations of input text for NLU, specifically ISO-15919 and IndicSOUNDEX, a custom SOUNDEX designed to work for Indic languages. We used a deep neural network based model to incorporate the information from alternate representations into the NLU model. We show that using alternate representations significantly improves the overall performance of NLU models when training data is limited.

## introduction building nlu models can be more challenging for languages that involve code-switching. recent times have seen a significant surge of interest in voice-enabled smart assistants, such as amazon's alexa, google assistant, and apple's siri. these assistants are powered by several components, which include automatic speech recognition (asr) and natural language understanding (nlu) models. the input to an nlu model is the text returned by the asr model. with this input text, there are two major tasks for an nlu model: 1) intent classification (ic), and 2) named entity recognition (ner). code-switching is a phenomenon in which two or more languages are interchanged within a sentence or between sentences. an example of code-switching from hindi is 'मे री shop-ping list म dove soap bar जोड़ ' ('add dove soap bar to my shopping list'). nlu models expect asr to return the text in the above form, which matches with the transcription of the training data for nlu models. then, nlu model would return the action as add items to shopping list, while it also recognizes 'dove soap bar' as the actual item name to be added to the shopping list. however, asr could inconsistently return 'मे री shopping list म dove soap बार जोड़ ', where the english word 'bar' is recognized as a hindi word 'बार'. note that while the hindi word 'बार' means something different than the english 'bar', their pronunciation is similar enough to mislead the asr model in the context of mixed-language utterance. in this case, the nlu model should become robust against such asr mistakes, and learn that 'dove soap बार' is equivalent to 'dove soap bar' in order to correctly recognize it as an item name. the phenomenon of codeswitching is common amongst other indic languages as well. asr inconsistencies can cause significant challenges for nlu models by introducing new data that the models were not trained on. such inconsistencies can occur due to 1) codeswitching in the data, especially when the input text contains more than one script (character set), 2) orthographic or morphological variations that exist in the input text, or 3) due to requirements for multilingual support. in a language such as english in the united states, where code-switching is less common, both asr and nlu models can perform quite well, as input data tend to be more consistent. however, when it comes to indic languages such as hindi, where code-switching is much more common, the question arises as to which representation of the input text would work best for nlu models. there is ongoing research on solving modeling tasks for data with code-switching. in geetha et al. (2018) , the authors explore using character and word level embeddings for ner tasks. some of this research focuses on using alternate representations of text for nlu modeling. for example, @xcite explore the use of cross-lingual transfer learning from english to japanese for ner by romanizing japanese input into a latin-based character set to overcome the script differences between the language pair. zhang and le-cun (2017) has shown using romanization for japanese text for sentiment classification hurt the performance of the monolingual model. in this paper, we explore the possibility of mitigating the problems related to asr inconsistency and code-switching in our input data by using two alternate representations of text in our nlu model: iso-15919 and in-dicsoundex. iso-15919 foot_0 was developed as a standardized latin-based representation for indic languages and scripts. soundex is an algorithm that provides phonetic-like representations of words. most work on soundex algorithms has focused primarily on monolingual solutions. one of the best known implementations with a multilingual component is @xcite ; however, it only identifies the language in which a given word is written to choose which pronunciation rules to apply. other attempts at multilingual soundex algorithms, particularly for indic languages, were smaller studies focused on two indic languages with or without english as a third language. @xcite developed a custom soundex algorithm for monolingual hindi and marathi word pair matching. @xcite describe an actual multilingual soundex implementation designed to cover hindi, gujarati, and english, which, in addition to the actual algorithm implementation, was aided by a matching threshold declaring two conversions a match even if they differed in up to two characters. in this study, we use iso-15919 and indic-soundex representations of text in a deep neural network (dnn) to perform multi-task modeling of ic and ner. we experiment on one high-resource indic language (hindi) and three low-resource indic languages (tamil, marathi, telugu). in section 2, we describe the two alternate representations of text that we explore. in section 3, we describe our data, model architecture, and detail our experimental setup. in section 4, we present our results followed by our conclusions in section 5. ## alternate representations of text using data transcribed in the original script of a language can cause problems for both monolingual and multilingual nlu models. for a monolingual model in a language where codeswitching, orthographic variations, or rich morphological inflections are common, nlu models may not be able to perform well on all the variations, depending on the frequency of these variations in the training data. for multilingual models, words with similar sound and meaning across different languages (e.g., loanwords, common entities) cannot be captured if the words are written in their original script. for example, the same proper noun 'telugu' is written as 'ते लु गु ' in hindi, as 'ெத ' in tamil, and as ' లుగు' in telugu. although they sound similar and mean the same thing, nlu model will see them as unrelated tokens if they are represented in their original scripts in the input data. from the nlu point of view, a text representation that can minimize variations of the same or similar words within a language and across different languages would be beneficial for both ic and ner tasks. in this section, we explore two alternative ways of text representations for indic languages: iso-15919 and a soundex-based algorithm, which we call indicsoundex. compared to using the original scripts, these two alternatives can represent the variants of the same word or root in the same way. for example, in the original hindi script (i.e., devanagari), the word for 'volume/voice' can be represented in two forms: 'आवाज़' and 'आवाज'. these two forms, however, are uniformly represented as 'āvāj' in iso-15919 and as the string 'av6' in indic-soundex. similarly, the english word 'list' may be transcribed as 'list' or as ' ' in telugu; however, they map to the same indic-soundex representation, 'ls8'. ## results to evaluate each model, we used our withheld test dataset and measured semer scores associated with the three different models. to obtain confidence intervals, we bootstrapped our test dataset by sampling ten times with replacement, and evaluated our models on each of these ten datasets. final semer was taken as the average of all the ten iterations. we used a student's t-test to calculate the significance of the improvements of the indicsoundex and iso-15919 models with respect to the baseline. here we present the results of our three models: the baseline dnn model, indicsoundex model, and iso-15919 model on each of the four languages. ## conclusion in this work, we explored the effect of using alternate representations of text for ic and ner models on a high-resource indic language (hindi) and three low-resource indic languages (tamil, telugu, marathi). we adopted a neural network based model architecture, where the alternate representations are incorporated in the embedding layers. based on the performance analysis over the baseline models, we saw that the alternate representations, while helping specific topics, do not help as much for the high-resource language overall. this is possibly due to the relatively high consistency in transcription and the existence of various tokens with similar meaning and sound in the training data. however, they helped significantly boost performance in the low-resource languages, thus creating potential applications in bootstrapping new languages quickly and cost-effectively. in the case of the low-resource languages we saw significant improvements overall when using either iso-15919 or indicsoundex. this suggests that smaller datasets stand to benefit from the use of alternative representations. based on tamil and marathi results, where indicsoundex performed better than base-line and iso-15919, we can conclude that the mitigation of the impact of multiple different scripts and inconsistent latin usage accomplished by indicsoundex seems to produce better results for our nlu models. the lack of impact of indicsoundex for telugu merits further investigation. our future work includes exploring different model architectures including transformer models, exploring these representations further by pre-training models with a combination of original tokens and alternate representations of tokens. we also plan to explore the use of character bigrams (of both original text and alternate representations of text) instead of unigram characters in the embedding layers. in indicsoundex is significant % change in iso-15919 is significant reservations -1.64% no -0.65% no books -0.43% no -5.18% no calendar -7.59% no -7.05% no callingandmessaging -4.36% no -0.08% no news +2.78% no +0.92% no photos -6.85% no -12.16% no media +7.79% no +11.13% yes global -2.89% yes -4.09% yes help +6.52% no +5.76% no smarthome -5.58% yes -4.32% yes questionandanswer +3.10% no +4.99% yes search -3.98% no +3.78% no music -0.45% no +2.18% yes notifications +2.74% no -2.49% no originalcontent -3.66% no -1.63% no recipes -0.44% no -0.39% no shopping +5.36% yes -0.82% no sports 0% no 0% no todolists -16.50% yes +1.08% no video -2.73% no -4% yes weather -3.89% no +16.36% yes overall -0.07% no +1.08% yes in indicsoundex is significant % change in iso-15919 is significant books -14.06% yes -18.28% yes calendar -8.52% yes -11.93% yes movietimes -7.91% yes -7.24% no callingandmssaging -12.44% yes -6.38% yes news -15.50% yes -6.14% yes media -12.23% yes 0% no global -4.20% yes -4.41% yes help -20.84% yes -17.72% yes smarthome -7.29% yes +0.05% yes search +7.15% yes +3.08% no music -8.03% yes -8.08% yes notifications -10.98% yes -6.81% yes originalcontent -9.92% yes -23.21% yes shopping -16.22% yes -13.44% yes sports +8.01% no -30.10% yes todolists -2.14% no +3.03% yes video -4.01% yes -14.65% yes weather -10.23% yes -10.91% yes overall -7.56% yes -6.23% yes in indicsoundex is significant % change in iso-15919 is significant books -6.95% yes -11.81% yes calendar +9.28% yes -0.11% yes movietimes -13.74% yes -11.03% yes callingandmessaging +3.09% yes 4.97% yes news +16.76% yes +1.90% yes media -24.50% yes -12.84% yes global +0.13% no -2.81% yes help -8.51% yes -0.76% no smarthome -14.36% yes -5.04% yes search -0.63% no -1.70% yes music -0.45% no -4.66% yes notifications +2.49% yes +5.50% yes originalcontent -22.84% yes +8.10% yes shopping +2.97% no +12.17% yes sports -38.48% yes -35.97% yes todolists +0.70% no -0.23% no video +5.25% yes +1.59% no weather +12.29% yes +32.18% yes overall -2.29% yes -1.47% yes in indicsoundex is significant % change in iso-15919 is significant books +2.42% no +0.37% no calendar +4.51% yes -10.33% yes movietimes -13.57% yes -8.83% yes callingandmessaging +5.41% yes +3.79% yes news +0.97% no +15.24% yes media -13.91% yes -0.61% no global +2.93% yes +3.57% no help +12.35% yes -0.45% no smarthome -0.15% no +0.31% no search -7.41% yes +2.90% yes music +1.02% no -4.55% yes notifications +1.33% no -4.15% no originalcontent -3.94% no +2.10% no shopping +0.55% no +1.59% no sports -4.43% no -14.31% no todolists +10.44% yes +10.05% yes video -0.72% no -0.97% no weather -25.98% yes -35.37% yes overall -0.21% no -2.42% yes