Zero-Shot Cross-Lingual Sequence Tagging as S eq2 S eq Generation for Joint Intent Classification and Slot Filling (2022)

Abstract: The joint intent classification and slot filling task seeks to detect the intent of an utterance and extract its semantic concepts. In the zero-shot cross-lingual setting, a model is trained on a source language and then transferred to other target languages through multi-lingual representations without additional training data. While prior studies show that pre-trained multilingual sequence-to-sequence (Seq2Seq) models can facilitate zero-shot transfer, there is little understanding on how to design the output template for the joint prediction tasks. In this paper, we examine three aspects of the output template – (1) label mapping, (2) task dependency, and (3) word order. Experiments on the MASSIVE dataset consisting of 51 languages show that our output template significantly improves the performance of pre-trained cross-lingual language models.

## introduction the joint intent classification and slot filling task is crucial for goal-oriented dialogue systems, seeking to detect the intent of an utterance and extract semantic concepts. this task has been widely studied in the literature @xcite @xcite . however, due to the difficulty of collecting and annotating large data sets, most studies focus on only a few highresource languages (e.g., english). to broaden the language coverage of models, zero-shot crosslingual transfer technique has been proposed @xcite @xcite . under the zero-shot cross-lingual setting, models are trained on a source language (e.g., english) with sufficient annotated training data and transfer to other target languages. in particular, recently, fitzgerald et al. (2022) show that pre-trained generative cross-lingual language models (xlms) @xcite 2021) can be applied to the joint intent classification and slot filling task. they formulate the joint task as sequence-to-sequence (seq2seq) generation, where the model generates the slot label for each word and the intent label for the utterance in a sequential manner based on an output template. however, the design of the output template is usually ad hoc and there is lack of understanding on how different template designs affect the performance of the zero-shot transfer. for example, in fig. 1 , the intent label "alarm set" can be represented by "set alarm". we found that the change of the surface form of the label significantly affects model's performance. in this paper, we examine three aspects in the design of output template, i.e. label mapping, task dependency, and word order. we found that all these aspects have significant influence on model performance. first, based on our observation on label mapping, we propose a concise hierarchical label mapping that leads to a better performance than the default label mapping used in annotation. second, we observed that generating the intent label before the slots labels leads to better performance. this is because intent classification is a relatively simpler task compared to slot filling and thus the correct task order prevents error propagation. finally, we found that word shuffle improves the diversity of data and therefore leads to better intent accuracy. experiments on the massive dataset @xcite consisting of 51 languages demonstrate that our proposed template design can significantly improve the performance of pre-trained generative xlms on this joint task in the zero-shot transfer setting. we also provide detailed ablation studies and discussion. we intend to release the source code for reproducing our experiments upon paper acceptance. ## method we first provide an overview of the seq2seq generation method with pretrained generative xlms for the joint intent classificaiton and slot filling task. then, we discuss the design of output template for seq2seq generation. ## experiment in this section, we evaluate our approach on a massive number of target languages. ## related work zero-shot cross-lingual joint intent classification and slot filling is crucial for developing goal-oriented dialogue systems for massive languages with less manually annotation @xcite @xcite . prior works on this joint task can be summarized into two lines. the first line follows a strict zero-shot setting, where only the data in source languages are used for training @xcite @xcite . the second line uses additional data consisting of words or utterances in target languages for training, where the additional data can be annotated data in target languages or synthetic data by code-switching and automatic translation @xcite @xcite . our work follows the strict zero-shot setting. prior works either formulate the joint task as sequence tagging and applies pretrained cross-lingual encoders @xcite to solve it @xcite @xcite , or formulate it as seq2seq generation and applies pretrained generative xlms to solve it @xcite . our work analyzes important variables in the output format of the seq2seq method. ## conclusion in this paper, we examine three variables of output format in seq2seq generation for zero-shot crosslingual joint intent classification and slot filling. experiments on the massive dataset consisting of 51 languages show that all the variables have significant influence on model performance. specifically, the output format should use a concise and hierarchical label mapping, and consider the label dependency carefully. 0.0 0.2 0.4 0.6 germanic semtic oghuz indo-aryan celtic hellenic romance indo-iranian finno-ugric armenian malayic japanese javanese karto-zan khmeric southern korean baltic mongolian lolo-burmese slavic albanian bantu south-central tai philippine vietic sinitic mt5 ours 0.0 0.1 0.2 0.3 0.4 0.5 in d o -a f r o -a s ia t ic t u r k ic u r a li c a u s t r o n e s ia n j a p o n ic k a r t v e li a n a u s t o a s ia t ic d r a v id ia n k o r e a n ic m o n g o li c s in o -t ib e t a n n ig e r -c o n g o k r a -d a i mt5 ours figure 5: exact match accuracy by language family. 0.0 0.1 0.2 0.3 0.4 0.5 -sov vso svo none mt5 ours figure 6: exact match accuracy by language order. 0.0 0.1 0.2 0.3 0.4 0.5 -binary none multiple avoidance mt5 ours figure 7: exact match accuracy by language politeness. 0.0 0.1 0.2 0.3 0.4 0.5 -singular and plural number neutral singular only none plural only mt5 ours figure 8: exact match accuracy by language imperative morphology. 0.0 0.1 0.2 0.3 0.4 0.5 -neither minimal maximal mt5 ours figure 9: exact match accuracy by language imperative hortative. 0.0 0.1 0.2 0.3 0.4 0.5 -present absent mt5 ours figure 10: exact match accuracy by language optative. 60 0.0 0.1 0.2 0.3 0.4 0.5 -special negative special imperative and negative normal imperative and negative special imperative mt5 ours figure 11: exact match accuracy by language prohibitive.