STIL - Simultaneous Slot Filling, Translation, Intent Classification, and Language Identification: Initial Results using m BART on M ulti ATIS ++ (2020)

Abstract: Slot-filling, Translation, Intent classification, and Language identification, or STIL, is a newly-proposed task for multilingual Natural Language Understanding (NLU). By performing simultaneous slot filling and translation into a single output language (English in this case), some portion of downstream system components can be monolingual, reducing development and maintenance cost. Results are given using the multilingual BART model (Liu et al., 2020) fine-tuned on 7 languages using the MultiATIS++ dataset. When no translation is performed, mBART’s performance is comparable to the current state of the art system (Cross-Lingual BERT by Xu et al. (2020)) for the languages tested, with better average intent classification accuracy (96.07% versus 95.50%) but worse average slot F1 (89.87% versus 90.81%). When simultaneous translation is performed, average intent classification accuracy degrades by only 1.7% relative and average slot F1 degrades by only 1.2% relative.

## introduction multilingual natural language understanding (nlu), also called cross-lingual nlu, is a technique by which an nlu-based system can scale to multiple languages. a single model is trained on more than one language, and it can accept input from more than one language during inference. in most recent high-performing systems, a model is first pre-trained using unlabeled data for all supported languages and then fine tuned for a specific task using a small set of labeled data @xcite . two typical tasks for goal-based systems, such as virtual assistants and chatbots, are intent classification and slot filling @xcite . though intent classification creates a language agnostic output (the intent of the user), slot filling does not. input 从盐湖城到加州奥克兰的航班 traditional output intent: flight slots: (盐湖城, fromloc.cityname), . . . (奥克兰, toloc.cityname), . . . (加州, toloc.statename) stil output intent: flight slots: (salt lake city, fromloc.cityname), . . . (oakland, toloc.cityname), . . . (california, toloc.statename) lang: zh instead, a slot-filling model outputs the labels for each of input tokens from the user. suppose the slot-filling model can handle l languages. downstream components must therefore handle all l languages for the full system to be multilingual across l languages. machine translation could be performed before the slot filling model at system runtime, though the latency would be fully additive, and some amount of information useful to the slotfilling model may be lost. similarly, translation could occur after the slot-filling model at runtime, but slot alignment between the source and target language is a non-trivial task @xcite . instead, the goal of this work was to build a single model that can simultaneously translate the input, output slotted text in a single language (english), classify the intent, and classify the input language (see contributions of this work include (1) the introduction of a new task for multilingual nlu, namely simultaneous slot filling, translation, intent clas-example input example output flüge von salt lake city nach oakland kalifornien @xmath0.city @xmath1.city @xmath2.city @xmath3.city @xmath4.state @xmath5.city @xmath6.city @xmath7.city @xmath8.city @xmath9.state @xmath10 ## dataset the airline travel information system (atis) dataset is a classic benchmark for goal-oriented nlu @xcite . it contains utterances focused on airline travel, such as how much is the cheapest flight from boston to new york tomorrow morning? the dataset is annotated with 17 intents, though the distribution is skewed, with 70% of intents being the flight intent. slots are labeled using the beginning inside outside (bio) format. atis was localized to turkish and hindi in 2018, forming multiatis @xcite , and then to spanish, portuguese, german, french, chinese, and japanese in 2020, forming multi-atis++ @xcite . in this work, portuguese was excluded due to a lack of portuguese pretraining in the publicly available mbart model, and japanese was excluded due to a current lack of alignment between japanese and english samples in multiatis++. hindi and turkish data were taken from multi-atis, and the training data were upsampled by 3x for hindi and 7x for turkish. prior to any upsampling, there were 4,488 training samples for english, spanish, german, french, and chinese. the test sets contained 893 samples for all languages except turkish, which had 715 samples. for english, spanish, german, french, and chinese, validation sets of 490 samples were used in all cases. given the smaller data quantities for hindi and turkish, two training and validation set configurations were considered. the first configuration matched that of @xcite , using training sets of 1,495 for hindi and 626 for turkish along with validation sets of 160 for hindi and 60 for turkish. in the second configuration, no validation sets were made for hindi and turkish (though there were still validation sets for the other languages), and the training sets of 1,600 hindi samples and 638 samples from multiatis were used. two output formats are considered, being (1) the non-translated, traditional case, in which translation of slot content is not performed, and (2) the translated, stil case, in which translation of slot content is performed. in both cases, the tokens, the labels, the intent, and the detected language are all output from the model as a single ordered text sequence, as shown in ## related work previous approaches for intent classification and slot filling have used either (1) separate models for slot filling, including support vector machines @xcite , conditional random fields @xcite , and recurrent neural networks of various types @xcite or (2) joint models that diverge into separate decoders or layers for intent classification and slot filling @xcite @xcite or that share hidden states @xcite . in this work, a fully text-to-text approach similar to that of the t5 model was used, such that the model would have maximum information sharing across the four stil sub-tasks. encoder-decoder models, first introduced in 2014 @xcite , are a mainstay of neural machine translation. the original transformer model included both an encoder and a decoder @xcite . since then, much of the work on transformers focuses on models with only an encoder pretrained with autoencoding techniques (e.g. bert by @xcite ) or auto-regressive models with only a decoder (e.g. gpt by @xcite ). in this work, it was assumed that encoder-decoder models, such as bart @xcite and t5 @xcite , are the best architectural candidates given the translation component of the stil task, as well as past state of the art advancement by encoder-decoder models on atis, cited above. rigorous architectural comparisons are left to future work. ## results and discussion results from the models are given in ## conclusion this preliminary work demonstrates that a single nlu model can perform simultaneous slot filling, translation, intent classification, and language identification across 7 languages using multiatis++. such an nlu model would negate the need for multiple-language support in some portion of downstream system components. performance is not irreconcilably worse than traditional slot-filling models, and performance is statistically equivalent with a small amount of additional training data. looking forward, a more challenging dataset is needed to further develop the translation compo-nent of the stil task. the english multiatis++ test set only contains 455 unique entity-slot pairs. an ideal future dataset would include freeform and varied content, such as text messages, song titles, or open-domain questions. until then, work remains to achieve parity with english-only atis models.