End-to-End Slot Alignment and Recognition for Cross-Lingual NLU (2020)

Abstract: Natural language understanding (NLU) in the context of goal-oriented dialog systems typically includes intent classification and slot labeling tasks. Existing methods to expand an NLU system to new languages use machine translation with slot label projection from source to the translated utterances, and thus are sensitive to projection errors. In this work, we propose a novel end-to-end model that learns to align and predict target slot labels jointly for cross-lingual transfer. We introduce MultiATIS++, a new multilingual NLU corpus that extends the Multilingual ATIS corpus to nine languages across four language families, and evaluate our method using the corpus. Results show that our method outperforms a simple label projection method using fast-align on most languages, and achieves competitive performance to the more complex, state-of-the-art projection method with only half of the training time. We release our MultiATIS++ corpus to the community to continue future research on cross-lingual NLU.

## introduction as a crucial component of goal oriented dialogue systems, natural language understanding (nlu) is responsible for parsing an utterance into a semantic frame to identify the user's need. these semantic frames are structured by what the user intends to do (the intent) and the arguments of the intent (the slots) @xcite . given the english example in figure 1 , we identify the intent of the utterance as "flight" and label the slots to extract the departure city and airline name. intent detection can be modeled as a sentence classification task where each utterance is labeled with an intent y i . slot filling is typically modeled as a sequence labeling task where given the utterance x 1...n , each word x i is labeled with a slot y i . despite the high accuracy achieved by neural models on intent detection and slot filling @xcite , training such models on a new language requires additional efforts to collect large amounts of training data. one would consider transfer learning from high-resource to low-resource languages to minimize the efforts of data collection and annotation. however, currently available multilingual nlu datasets @xcite only support three languages distributed in two language families, which hinders the study of cross-lingual transfer across a broad spectrum of language distances. in this paper, we release a new multilingual nlu corpus that contains training, development, and test data for six new languages in addition to the three languages in the multilingual atis corpus @xcite . the resulting corpus, namely multiatis++, consists in total of 37,084 training examples and 7,859 test examples covering nine languages in four language families. using our corpus, we explore the use of multilingual bert encoder @xcite , machine translation (mt), and label projection methods for multilingual training and cross-lingual transfer. furthermore, we propose an end-to-end model for joint slot label alignment and recognition, so that it no longer relies on slot label projections using external word alignment tools @xcite or engineered features @xcite , which may not generalize well to low-resource languages. our model performs soft label alignment using an attention module which is trained jointly with other model components on intent classification, slot filling, and an augmented reconstruction objective designed to improve the soft label alignment. experimental results show that our method uses the same amount of training time as a simple label projection approach using fast-align, while achiev-ing significantly higher slot f1 on most languages. furthermore, our method achieves competitive performance to the more complex, state-of-the-art label projection method that uses linguistic features to improve projection quality, while using half of the training time. finally, our results show different trends when comparing various cross-lingual transfer methods on different languages, which emphasizes the need to evaluate cross-lingual transfer methods on a diverse set of languages to fully illustrate the strengths and weaknesses of each method. ## related work cross-lingual transfer learning has been studied on a variety of sequence tagging tasks including part-of-speech tagging @xcite @xcite , named entity recognition @xcite @xcite , and natural language understanding @xcite @xcite . existing methods can be roughly categorized into two categories: transfer through multilingual models and transfer through machine translation. transfer via multilingual models for closely related languages with similar alphabets, it is beneficial to train a multilingual model with shared character encoder to learn common character-based features @xcite . however, such techniques are less effective when applied to dissimilar languages that lack common lexical features. @xcite focus on the multisource transfer scenario and apply adversarial training to extract language-invariant features shared by source languages. recent advances on crosslingual representations have enabled transfer between dissimilar languages. representations from multilingual neural machine translation (nmt) encoders have been shown to be effective for crosslingual text classification @xcite @xcite . in this work, we use multilingual bert @xcite , an unsupervised cross-lingual language model trained on monolingual texts from a wide range of languages and has been shown to provide powerful sentence representations that lead to promising performance for zero-resource cross-lingual language understanding tasks @xcite . we leave the use of other recently proposed cross-lingual language models such as xlm-r @xcite to future work. transfer via machine translation requires translating the source language training data into the target language or translating the target language test data into the source language. despite its empirical success on cross-lingual text classification tasks @xcite , it faces a challenging problem on the sequence tagging tasks: labels on the source language sentences need to be projected to the translated sentences. most of the prior work relies on unsupervised word alignment from statistical mt @xcite @xcite or attention weights from nmt models @xcite . other heuristic approaches include matching tokens based on their surface forms @xcite @xcite and more complex projection approaches that combine linguistic features with information from the mt systems @xcite . by contrast, our method does not rely on external word alignment or linguistic features, but models label projection through an attention module that can be jointly trained with other model components on the machine translated data. ## data one of the most popular datasets for multilingual nlu is the atis dataset @xcite and its multilingual extension @xcite . the atis dataset is created by asking each participant to interact with an agent (who has access to a database) to solve a given air travel planning problem. @xcite extend the english atis to hindi and turkish by manually translating and annotating a subset of the training and test data via crowdsourcing. 1 to facilitate study on cross-lingual transfer across a broader spectrum of language distances, we create the multiatis++ corpus by extending both the training and test set of the english atis corpus to six additional languages. 2 the resulting corpus covers nine languages in four different language families including indo-european (english, spanish, german, french, portuguese, and hindi), sino-tibetan (chinese), japonic (japanese), and altaic (turkish). for each new language, we hire professional native translators to translate the english utterances and annotate the slots at the same time. when en show departures from atlanta for american o o o b-fromloc.city_name o b-airline_name es muestra salidas desde atlanta de american o o o b-fromloc.city_name o b-airline_name pt mostre partidas de atlanta da american o o o b-fromloc.city_name o b-airline_name de zeige abflüge von atlanta für american o o o b-fromloc.city_name o b-airline_name fr montrer des départs d' atlanta pour american o o o o b-fromloc.city_name o b-airline_name zh o b-airline_name o b-fromloc.city_name o ja b-fromloc.city_name o b-airline_name o hi अमे $रकन के (लए अटलां टा से /0थान 2दखाएं b-airline_name o o b-fromloc.city_name o o o tr atlanta ' dan american kalkislarini goster b-fromloc.city_name o o b-airline_name o o figure 1: an english training example and its translated versions in the multiatis++ corpus. the english utterance is manually translated to the other eight languages including spanish (es), portuguese (pt), german (de), french (fr), chinese (zh), japanese (ja), hindi (hi), and turkish (tr). for each language, we show the utterance followed by the slot labels in the bio format. the intent of the utterances is the flight intent. we show an english training example and its translated versions in the other eight languages in figure 1 and report the data statistics in ## multilingual nlu in our first set of experiments, we explore using pretrained multilingual bert encoder for multilingual nlu. we compare the following training strategies to leverage the full supervised training data: setup we train the models using the adam optimizer (kingma and ba, 2015) for 20 epochs and select the model that performs the best on the development set (details in appendix a). following @xcite , we use intent accuracy and slot f1 as evaluation metrics. results comparison with sota on english atis, qin et al. (2019) report 97.5% intent accuracy and 96.1% slot f1 when using bert with their proposed stack-propagation architecture. this is comparable to our target only with bert scores in ## cross-lingual transfer in this section, we compare the following methods for cross-lingual transfer where we only use the english training data and a small amount (fewshot) or no (zero-shot) training data from the target language: for all three mt-based approaches, we use aws translate for automatic translation and perform multilingual training on the english and machine translated data. we adopt the same setup as the previous section, except that we select the model at the last epoch as we assume no access to the development set from the target language in this setting. for the attention module in our soft-alignment model, we set the temperature τ = 0.1. ## conclusion we introduce multiatis++, a multilingual nlu corpus that extends the multilingual atis corpus to nine languages across four language families. we use our corpus to evaluate various cross-lingual transfer methods including the use of multilingual bert encoder, machine translation, and label projection. we further introduce a novel end-to-end model for joint slot label alignment and recognition that requires no external label projection. experiments show that multilingual bert brings substantial improvements on multilingual training and cross-lingual transfer tasks. furthermore, our model outperforms the simple projection baseline using fast-align on most languages, and achieves competitive performance to the state-of-the-art label projection approach with only half of the training time. we release our multiatis++ corpus to facilitate future research on cross-lingual nlu to bridge the gap between cross-lingual transfer and supervised methods.