AISFG : Abundant Information Slot Filling Generator (2022)

Abstract: As an essential component of task-oriented dialogue systems, slot filling requires enormous labeled training data in a certain domain. However, in most cases, there is little or no target domain training data is available in the training stage. Thus, cross-domain slot filling has to cope with the data scarcity problem by zero/few-shot learning. Previous researches on zero/few-shot cross-domain slot filling focus on slot descriptions and examples while ignoring the slot type ambiguity and example ambiguity issues. To address these problems, we propose Abundant Information Slot Filling Generator (AISFG), a generative model with a novel query template that incorporates domain descriptions, slot descriptions, and examples with context. Experimental results show that our model outperforms state-of-the-art approaches in zero/few-shot slot filling task.

## introduction slot filling is a critical part of downstream tasks in natural language understanding (nlu) such as dialogue systems. recently, some supervised slot filling models have achieved state-of-the-art results within deep learning @xcite @xcite .nonetheless, these methods have a strong dependency on the domain-specific labels, which is not capable of transferring to new domains that always contain little or no data. to alleviate the problem of resource gap between source and target domains, cross-domain zero-shot has become an important research direction in slot filling. however, most researches on slot filling utilize token-level classification frameworks, which means they either convert it to a bio tag labeling task @xcite @xcite or predict the start and end position as qa task in the sentence @xcite , to extract spans. recently, prompt learning methods reformulate the downstream tasks to a similar form with the pre-training tasks, which can fully utilize the knowledge encoded in plms and improve the performance of the downstream tasks in the scenarios of data scarcity. motivated by this, we consider making the slot filling task consistent with plms pre-training tasks. in this paper, we propose a generative templatebased zero-shot slot filling framework named abundant information slot filling generator (aisfg), which utilizes pre-trained generative model as the backbone and generates responses in natural language style. thus, the slot filling task is consistent with the pre-training task of the plm. in particular, we notice that shared cross-domain slot types sometimes refer to totally different entities across different domains. for example, in the snips dataset @xcite , domains ratebook and searchscreeningevent share a common slot type object_type, but it refers to book type and movie schedule, respectively. we call this slot type ambiguity issue. moreover, we argue that incorporating slot examples only is far from fully utilizing the example information. for instance, for sentence give 5 out of 6 stars to creatures of light and darkness, we give the slot description to find best rating with examples like 6 and 5, the model produces the wrong answer 5 but not 6. we conjecture that, the model just learn that predicting a number is satisfying from these two examples but does not understand what is the best rating means. we call this example ambiguity issue. thus, we attach our attention to domain-specific descriptions and examples with context to alleviate the above two issues. specifically, we design the query template by incorporating domain descriptions, slot descriptions and examples with context. the contribution of this paper can be summarized in three aspects: • we propose a generative template-based zeroshot slot filling framework. to the best of our knowledge, we are the first to apply the generative framework to perform the zero-shot cross-domain slot filling task. ## related work the main challenge of cross-domain slot filling is to handle domain-specific slot types which have few or no supervision signals during the training stage. to handle the unseen slots, previous methods introduce slot descriptions @xcite @xcite and slot examples @xcite to capture the semantic relationship between unseen slots and input sentences. based on these researches, we incorporate more information, like domain description and example context, to boost the performance on unseen slots. typically, traditional methods formulate the slot filling task as a token-level classification task @xcite @xcite . to leverage the knowledge encoded in pre-trained language models, rcsf @xcite utilizes bert @xcite to predict the start/end position of the target slot entity. however, the downstream task (i.e., token/position prediction) is inconsistent with the pre-training task (i.e., masked lm and nsp for bert), which may limit the expressiveness of the plm @xcite . to bridge the gap, we treat the slot filling as a generative task and utilize a plm whose pre-training task is also a generative task. ## conclusion & discussion in this paper, we introduce a novel zero-shot crossdomain slot filling model named aisfg, which can adapt to unseen domains seamlessly with the help of domain and slot description and together with full context examples. experiments show that our model significantly outperforms existing zero-shot cross-domain slot filling approaches. moreover, we conduct error analysis and case study to better understand our proposed model and leave solving boundary mistakes and vocabulary mistakes as our future work. liwen wang, xuefeng li, jiachi liu, keqing he, yuanmeng yan, and weiran xu. 2021. bridge to target domain by prototypical contrastive learning and label confusion: re-explore zero-shot learning for slot filling. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 9474-9480, online and punta cana, dominican republic. association for computational linguistics. kaisheng yao, geoffrey zweig, mei-yuh hwang, yangyang shi, and dong yu. 2013. recurrent neural networks for language understanding. in in-terspeech 2013, 14th annual conference of the international speech communication association, lyon, france, august 25-29, 2013, pages 2524-2528. isca. mengshi yu, jian liu, yufeng chen, jinan xu, and yujie zhang. 2021a. cross-domain slot filling as machine reading comprehension. in proceedings of the thirtieth international joint conference on artificial intelligence, ijcai-21, pages 3992-3998. international joint conferences on artificial intelligence organization. main track. mengshi yu, jian liu, yufeng chen, jinan xu, and yujie zhang. 2021b. cross-domain slot filling as machine reading comprehension. in proceedings of the thirtieth international joint conference on artificial intelligence, ijcai 2021, virtual event / montreal, canada, 19-27 august 2021, pages 3992-3998. ijcai.org. a appendix a.1 domain descriptions domain domain description addtoplaylist add to playlist bookrestaurant book restaurant getweather get weather playmusic play music ratebook rate book searchcreativework search creative work searchscreeningevent search screening event