From Masked Language Modeling to Translation: Non- E nglish Auxiliary Tasks Improve Zero-shot Spoken Language Understanding (2021)

Abstract: The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce xSID, a new benchmark for cross-lingual (x) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation transfer works best for intent classification.

## introduction digital conversational assistants have become an integral part of everyday life and they are available, e.g., as standalone smart home devices or in smartphones. key steps in such task-oriented conversational systems are recognizing the intent of a user's utterance, and detecting the main arguments, also called slots. for example, for an utterance like "add reminder to swim at 11am tomorrow", these key natural language understanding (nlu), or spoken language understanding (slu) tasks are illustrated in figure 1 . as slots depend on the intent type, leading models typically adopt joint solutions @xcite . despite advances in neural modeling for slot and intent detection ( § 6), datasets for slu remain limited, hampering progress toward providing slu for many language varieties. most avail-add reminder to swim at 11am tomorrow intent: add reminder figure 1 : english example from xsid annotated with intents (add reminder) and slots ( todo , datetime ). the full set of languages is shown in most previous efforts to multilingual slu typically focus on translation or multilingual embeddings transfer. in this work, we propose an orthogonal approach, and study non-english auxiliary tasks for transfer. we hypothesize that jointly training on target language auxiliary tasks helps to learn properties of the target language while learning a related task simultaneously. we expect that this helps to refine the multilingual representations for better slu transfer to a new language. we evaluate a broad range of auxiliary tasks not studied before in such combination, exploiting raw data, syntax in universal dependencies (ud) and parallel data. our contributions i) we provide xsid, a new cross-lingual slu evaluation dataset covering arabic (ar), chinese (zh), danish (da), dutch (nl), english (en), german (de), indonesian (id), italian (it), japanese (ja), kazakh (kk), serbian (sr), turkish (tr) and an austro-bavarian german dialect, south tyrolean (de-st). ii) we experiment with new @xcite . ## models our main hypothesis is that we can improve zeroshot transfer with target-language auxiliary tasks. we hypothesize that this will help the multilingual pre-trained base model to learn peculiarities about the target language, while it is learning the target task as well. to this end, we use three (sets of) tasks with a varying degree of complexity and availability: 1) masked language modeling (mlm): which is in spirit similar to pre-training on another domain @xcite , however, we learn this jointly with the target task to avoid catastrophic forgetting @xcite ; 2) neural machine translation (nmt): where we learn english slu as well as translation from english to the target language; and 3) universal dependency (ud) parsing: to insert linguistic knowledge into the shared parameter space to learn from syntax as auxiliary task besides learning the slu task. in the following subsections, we first describe the implementation of our baseline model, and the machine translation-based model, and then de7 for more details on this procedure, we refer to scripts/0.fixorigannotation.py in the repo. scribe the implementation of all auxiliary tasks (and the data used to train them). auxiliary tasks are sorted by dataset availability (mlm nmt ud), where the first type can be used with any raw text, the second one needs parallel data -which is readily available for many languages as a byproduct of multilingual data sources -and the last one requires explicit human annotation. for south tyrolean, a german dialect, no labeled target data of any sort is available; we use the german task data instead. we provide more details of data sources and sizes in appendix b. ## related work for related datasets, we refer to § 2.1; in this section we will discuss different approaches on how to tackle cross-lingual slu. work on cross-lingual slu can broadly be divided into two approaches, whether it is based mainly on parallel data or multilingual representations. the first stream of research focuses on generating training data in the target language with machine translation and mapping the slot labels through attention or an external word aligner. the translation-based approach can be further improved by filtering the resulting training data @xcite , post-fixing the annotation by humans @xcite , or by using a soft-alignment based on attention, which alleviates error propagation and outperforms annotation projection using external word aligners @xcite . the second stream of research uses multilingual representations. upadhyay et al. (2018 al. (2019b) propose refining the alignment of bilingual word representations. the best performing variants use contextualized bert variants @xcite , which we depart from. we propose a third, orthogonal line of research: joint target-language auxiliary task learning. we hypothesize that jointly training on target language auxiliary tasks helps to learn properties of the target language while learning a related task simultaneously. we frame masked language modeling, universal dependency parsing and machine translation as new auxiliary tasks for slu. some work on slu showed that syntax in graph convolution networks is beneficial for slots @xcite . contemporary work shows that highresource english data helps target language modeling in sequential transfer setups @xcite . we focus on non-english target data for joint slu in a single cross-lingual multi-task model instead. ## conclusions we introduced xsid, a multilingual dataset for spoken language understanding with 13 languages from 6 language families, including an unstudied german dialect. xsid includes a wide variety of intent types and homogenized annotations. we propose non-english multi-task setups for zero-shot transfer to learn the target language: masked language modeling, neural machine translation and ud parsing. we compared the effect of these auxiliary tasks in two settings. our results showed that masked language modeling led to the most stable performance improvements; however, when a language is not seen during pre-training, ud parsing led to an even larger performance increase. on the intents, generating target language training data using machine translation was outperforming all our proposed models, at a much higher computational cost however. our analysis further shows that nmt-transfer struggles with span detection. given training time and availability trade-off, mlm multitasking is a viable approach for slu.