Bootstrapping Multilingual Intent Models via Machine Translation for Dialog Automation (2018)

Abstract: With the resurgence of chat-based dialog systems in consumer and enterprise applications, there has been much success in developing data-driven and rule-based natural language models to understand human intent. Since these models require large amounts of data and in-domain knowledge, expanding an equivalent service into new markets is disrupted by language barriers that inhibit dialog automation. This paper presents a user study to evaluate the utility of out-of-the-box machine translation technology to (1) rapidly bootstrap multilingual spoken dialog systems and (2) enable existing human analysts to understand foreign language utterances. We additionally evaluate the utility of machine translation in human assisted environments, where a portion of the traffic is processed by analysts. In English→Spanish experiments, we observe a high potential for dialog automation, as well as the potential for human analysts to process foreign language utterances with high accuracy.

## introduction with the present advances in natural language understanding and speech recognition technologies, unprecedented opportunities have been created for realizing natural and sophisticated humanmachine conversations to accomplish routine tasks. there has been a resurgence of speech/textbased conversation systems spanning multiple c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. for practical applications, such conversational agents and speech/text analytics systems, the meaning of a sentence may be approximated as one or more actionable labels, or intents, associated with the input utterance. in such cases, the natural language understanding (nlu) task is modeled as an intent classification problem. although ambiguity is present in natural language, data-driven nlu systems have been successful in modeling user intents in many application domains. many commercial and enterprise applications service customers from different geographic locations and varying language proficiencies, requiring multilingual nlu for human-machine interaction. in order to deploy an intent classification system for a new language a new set of labeled training data is conventionally required. this data is often unavailable before a solution is deployed, instead requiring a human-driven dialog system depending on intent analysts or live agents. in time, a sufficient amount of production data may be collected to build a data-driven intent model; however this approach is expensive to operate and ignores valuable knowledge present in other languages that could be used to build an initial model. in this paper, we evaluate the use of machine translation (mt) as a tool to bridge the knowledge present in one or more intent models for the creation of an intent classifier in a target language. given mt's capability to translate the content of utterances in a source language to a target language, our goal is to minimize the number of language proficient intent analysts needed to support a production-scale multilingual dialog system in the absence of target language data. the remainder of the paper is organized as follows: in section 2 we discuss the details of the intent classification system and present the nlu model, including our two mt architectures in a multilingual spoken dialog system. in section 3 we outline our experiment and describe our asr, mt, and nlu models. in sections 4-7, we evaluate the asr, mt, and nlu model performance. in section 8 we evaluate human agents' ability to label the intents of translated user utterances and summarize our findings in section 10. ## nlu for customer care a single utterance is tagged with three types of labels: intents, entities, and conversational handlers. intents are domain-specific labels such as sales, tech assistance, and billing. entity labels represent the names of products or services mentioned by the user. these include specific models of smart phones or subscription services. conversational handlers are labels which are similar to speech acts to guide the conversation. for example, live agent, confused, or foreign lan-guage. in our experiments, intents and entities are labeled as "session variables" (sv), while conversation handles are partitioned into "task names" (tn) and "event names" @xcite . examples of each in our experiments are shown in a joint svm classifier is trained by concatenating the tn, en, and sv labels into a unique label. the model comprises a set of binary svm classifiers, with each classifier predicting if the input is assigned or not assigned to a particular label type. for a given input utterance x, the joint label is computed as: y * = arg max @xmath0,sv,en f y (x, y). the feature set f may comprise application context, conversation context, and the utterance. we use n-gram word-level features in this experiment. ## experimental setup we evaluate the efficacy of bootstrapping a spanish intent classifier using the data and underlying models from an english spoken language dialog system. the data set consists of customer voice responses to the message "how may i help you?" in the customer's native language at the beginning of a phone call to an interactive voice response (ivr) system. we assume that no spanish intent data is available during training time and evaluate the performance of our bootstrapped spanish models against an intent model trained with a standard training set. ## asr performance we use sclite from the nist speech recognition scoring toolkit 2 to compute the word error rate (wer) and utterance error rate. after further clean-up and adjudication, we observe a 33.2% wer, with 60.0% of the utterances containing errors (32% substitutions, 22% deletions, 28% insertions). a majority of the substitution errors were confusions between singular and plural (e.g. @xmath1.g. @xmath2. other errors included phonetic confusions (e.g. @xmath3, named entity misrecognitions (e.g. @xmath4, and a high frequency of dropped articles (e.g. de, la, a) caused by speaker under-articulation. of these types of errors, the most detrimental are substitutions of named entities, verbs, and nouns that are not lemmatization errors. another issue driving up the wer score caused by the ivr system truncating audio longer than four seconds to reduce latency. ## machine translation quality in order to assess the translation quality, we postedit the translations of the human-transcribed utterances and report the case-insensitive bleu 2 https://www.nist.gov/itl/iad/mig/tools and ter scores in the primary sources of errors are incomplete sentences, lack of punctuation; lexical mistranslations of key words: (e.g. equipo (equipment)→team; dirección (address)→direction; reclamo (complaint)→ claim); ambiguous translations (e.g. fac-@xmath5; and duplicated words during translation (e.g. payment @xmath6. many of these issues are due to lack of in-domain mt training data and low tolerance of asr errors. highly repetitive errors indicate that an automatic post-editing system could substantially improve the translation system's quality. ## native nlu performance our reference native spanish intent classification model is trained on asr outputs since an insufficient amount of human-transcribed intent modeling data is available. from evaluating nlu performance on human transcripts, we observe improvements that rival human labeling performance at 10% rejection and above. at 20% rejection, the model significantly outperforms human labeling. ## bootstrapped nlu performance while the native spanish model has a 13.5% error rate at 10% rejection when processing asr hypotheses, the bootstrapped models have double the error rate due to the use of out-of-domain machine translation. on asr hypotheses, the @xmath9.7%, while the @xmath10.5% at 10% rejection. to better understand how machine translation further corrupts the bootstrapped @xmath11, we compare the errors it makes to the native spanish model. in ## analyst performance on translated text finally, we measure ia labeling performance on translated utterances. in our conventional scenario, intent analysts listen to audio segments in their native language and provide an intent label. instead, we replace the original audio with machine translated or translation post-edits of asr hypotheses. fig. 3 provides error-rejection curves for ias, with error rates at 0%, 10% and 20% rejection reported in ## related work the use of mt to translate texts in other languages into english for sentiment analysis was proposed in @xcite . @xcite show that sometimes mt performs inadequate translations on essential parts of a text, affecting sentiment analysis performance. our results confirm this phenomena due to a lack of in-domain mt training data. @xcite explore learning multilingual sentence embeddings with neural mt, which can aid in multilingual search. prior to that, multilingual approaches leveraged lexical resources such as multiwordnet @xcite to bridge concepts from one language to another. ## conclusions we have executed an experiment to measure machine translation's ability to rapidly bootstrap intent classification models for new languages. in our @xmath15, we observe that although the initial results appear to be substantially worse than a native spanish intent classification model, we show that mt can provide a degree of automation that supports human-assisted multilingual dialog systems that can be deployed to production on day one, reducing the need for human agent support over a fully manual solution. there is further promise that model improvements can be obtained by improving the asr and machine translation models to include in-domain data. finally, we observe it is better to use the on-line @xmath16.