Generative Zero-Shot Prompt Learning for Cross-Domain Slot Filling with Inverse Prompting (2023)

Abstract: Zero-shot cross-domain slot filling aims to transfer knowledge from the labeled source domain to the unlabeled target domain. Existing models either encode slot descriptions and examples or design handcrafted question templates using heuristic rules, suffering from poor generalization capability or robustness. In this paper, we propose a generative zero-shot prompt learning framework for cross-domain slot filling, both improving generalization and robustness than previous work. Besides, we introduce a novel inverse prompting strategy to distinguish different slot types to avoid the multiple prediction problem, and an efficient prompt tuning strategy to boost higher performance only training fewer prompt parameters. Experiments and analysis demonstrate the effectiveness of our proposed framework, especially huge improvements (+13.44% F1) on the unseen slots.

## introduction slot filling in a task-oriented dialogue system aims to extract task-related information like hotel_name, hotel_address from user queries, which is widely applied to existing intelligent conversation applications @xcite . traditional supervised methods @xcite @xcite @xcite have shown remarkable performance, but they still rely on large-scale labeled data. lack of generalization to new domains hinder its further application to practical industrial scenarios. in this work, we focus on zero-shot cross-domain slot filling which transfers knowledge from the source domain d s to the target domain d t without requiring any labeled training data of d t . conventional approaches @xcite @xcite formulate slot filling as a sequence labeling task and use meta-information such as slot descriptions and slot examples to capture the semantic relationship between slot types and input tokens. however, these models only learn a surface mapping of the slot types between d s and d t and get poor performance on unseen slots in the target domain @xcite . further, @xcite @xcite propose a machine reading comprehension (mrc) framework for slot filling to enhance the semantic interaction between slot types and slot values. they firstly construct many well-designed question templates based on slot schema or slot examples, then train an mrc model @xcite to predict corresponding slot values for a given slot type question. but they rely on handcrafted question templates using heuristic rules and pre-defined ontologies, which suffers from poor model robustness. besides, employing additional pre-training on large-scale external mrc datasets is also timeconsuming and prohibitively expensive. normal input none answer ilse delange postive ilse negative answer inverse prompting what is the _____? {playlist, artist, ...} add ilse delange to my journey playlist label prompt journey my journey playlist none music_item entity_name none entity_ entity_ entity name playlist none ilse delange journey all slot types original sentence template main task ... ... ... artist artist ... what is the _____? {playlist, artist, ...} add ilse delange to my journey playlist all slot types original sentence template plms ... to solve the above issues, in this paper, we propose a generative zero-shot prompt learning (gzpl) framework for cross-domain slot filling. instead of transforming the slot filling task into sequence labeling or mrc, we formulate it as a language generation task (see fig 1 ). specifically, we concat the question of each slot type, names of all slot types, the input query together to construct the input sequence and take the related slot values as output sequence. the converted text-to-text format has two benefits for zero-shot slot filling: our contributions are three-fold: (1) we propose a simple but strong generative zero-shot prompt learning framework for cross-domain slot filling, which has better generalization capability and robustness than previous work. (2) we present a novel inverse prompting strategy to distinguish different slot types to avoid the multiple prediction problem. besides, we introduce an efficient prompt tuning strategy to boost higher performance only training fewer prompt parameters. (3) experiments and analysis demonstrate the effectiveness of our proposed framework, especially for good generalization to unseen slots (f1 +13.44% ↑), strong robustness to different templates (∆ f1 +10.23% ↑), parameter efficiency (10x fewer parameters). ## methodology our model is shown in fig 2 . in our framework, we first construct several simple template sentences for the model input, where each sentence includes a slot type question, all slot types and the original query. then we use a plm to generate the corresponding slot values. along with the main task formulation, we perform an inverse-prompting task to warm up the parameters to strengthen the relationship between entities and slot types. ## conclusion in this paper, we introduce a generative prompt learning framework for zero-shot cross-domain slot filling. based on this, we introduce the label prompt strategy and the inverse prompting to improve the generalization capability and robustness of the framework. another prefix-tuning mechanism is performed to boost model training efficiency. the exhaustive experimental results show the effectiveness of our methods, and the qualitative analysis inspire new insight into related area. generally, our framework can be applied to more complex situations, such as nested ner, discontinuous/multiple slots, which we leave to future work. another interesting direction is to improve the inference efficiency, like concat all the slot questions together and get final results.