C5L7 : A Zero-Shot Algorithm for Intent and Slot Detection in Multilingual Task Oriented Languages (2022)

Abstract: Voice assistants are becoming central to our lives. The convenience of using voice assistants to do simple tasks has created an industry for voice-enabled devices like TVs, thermostats, air conditioners, etc. It has also improved the quality of life of elders by making the world more accessible. Voice assistants engage in task-oriented dialogues using machine-learned language understanding models. However, training deep-learned models take a lot of training data, which is time-consuming and expensive. Furthermore, it is even more problematic if we want the voice assistant to understand hundreds of languages. In this paper, we present a zero-shot deep learning algorithm that uses only the English part of the Massive dataset and achieves a high level of accuracy across 51 languages. The algorithm uses a delexicalized translation model to generate multilingual data for data augmentation. The training data is further weighted to improve the accuracy of the worst-performing languages. We report on our experiments with code-switching, word order, multilingual ensemble methods, and other techniques and their impact on overall accuracy.

## introduction task-oriented languages have become standard in voice-enabled devices and voice assistants. while there has been extensive research on task-oriented dialogue systems in limited domains, most of these systems are built in a limited set of languages due to a lack of labeled multilingual corpus. amazon's massive dataset is a new resource for taskoriented language understanding that has 996k utterances annotated with intent and slot labels, along with their translations into 51 languages. the mas-sive dataset is a unique resource for conducting multilingual language understanding research, and in particular building zero-shot learning algorithms where using only one language data, the trained system can perform language understanding tasks in the rest of the unseen languages. the importance of such training algorithms cannot be understated -labeled data is expensive and time-consuming to generate and hence any approach that reduces the cost and time to train such a multilingual system is desirable. there are numerous hurdles in creating a zeroshot multilingual language understanding system. while machine translation systems can be used for translating utterances and creating a parallel corpus for training, aligning slot labels across languages can be challenging. in addition, if we expect the multilingual model representation to leverage information across languages, the input text representation needs to have the same tokenization process across languages. furthermore, low-density languages are hard to get open-source resources for. in this paper, we first review the related work in section 2. next, we address the issues listed above by introducing a novel delexicalized annotated utterance translation algorithm that is described in section 3. to align code representations across languages, we randomly switched the language for a small percentage of the words. finally, we explored the possible impact of using all the utterance translations instead of just one utterance in a specific language and were surprised by the accuracy boost. these and other experiments and analyses are described in section 4. ## related work transformer-based large multilingual masked language models, such as mbert @xcite , xlm (lam-ple and conneau, 2019) xlm-r @xcite , and mt5 @xcite , have prevailed in crosslingual language understanding. these models are pre-trained on a large multilingual text corpus to create a language representation that allows crosslingual transfer on down-streaming tasks, such as cross-lingual document classification @xcite , role labeling @xcite , question answering @xcite @xcite and named entity recognition @xcite . in the field of natural language understanding, liu and lane and chen et al. trained for intent classification and slot-filling tasks jointly to learn the inherent correlation between the two tasks via multitask learning. castellucci et al. further used a joint bert-based model to detect intents and extract slots for the multi-lingual scenario including english and italian languages. however, systematic work on massive languages datasets (51 locales including sufficient variance of language order types including subject-initial, verb-initial and no preferred word order) has not been paid enough attention until now due to the lack of labeled datasets. with the availability of the massive dataset @xcite with annotation for slot-filling and intent classification, and virtual assistant evaluation metrics and scoring tools, we will able to push the state of the art of multilingual natural language processing for a task-oriented dialogue system @xcite . to tackle the difficulty/high cost of collecting low-resource language data previous work @xcite @xcite explored the use of machine translation to get translated data @xcite and utilize zero-shot learning @xcite to transfer the understanding learned on one language to another language. however, the correspondence across all languages in terms of intent and slot alignment is insufficiently incorporated into the training and inference phases of the cross-lingual nlu model. in this paper, we explore how to represent connection among massive languages in the model. besides, inspired by the common code-switch behavior and multilingual speakers and previous work on learning cross-lingual structure @xcite @xcite , we further explore the use of code-switch and delexicalization as anchor points to bridge the transfer learning among languages. ## discussion and future work how good is the delexicalized slot translation? one approach to quantify this would be to generate an annotated translation from english to language i using google translator and then translate it back to english and then compute a bleu score. our zero-shot ensemble method using generated parallel corpus gives us better intent accuracy than the baseline full-set result in @xcite . however, the slot accuracy is still much lower. one of the reasons could be that the slot values don't translate well to other languages. for example, a christian name is not something that will be common in chinese data. using languagespecific values probably will yield better results. the ensemble method in a real-world setting requires us to translate utterance t in language i, to all other 50 languages. this requires to generate n 2 translation, which is expensive on google translate. for our experiment, we instead used the translations from english. one issue with this approach could be that english-to-target language translation might be of better quality than the translation of input language to a target language. doing the full experiment will be conclusive. another drawback of the ensemble approach is the need for n real-time translations and n parallel real-time runs. however, one way to reduce this complexity is to find a small subset of languages that we can use for voting purposes. ## conclusion we presented a zero-shot, multilingual, joint intentdetection and slot-filling algorithm based on xlm-r transformer and amazon's massive dataset. we showed that our delexicalized translation approach to generating a parallel corpus for data augmentation is a viable approach for training zeroshot algorithms. we showed that training using data from all language order types gives superior accuracy than using only a single language order type data in most cases -n massive data, the svo category performed equally well when using just the svo subset. furthermore, our experiments showed that using an ensemble approach with translations of the input utterance can lead to a significant gain in accuracy.