Zero-shot Cross-domain Dialogue State Tracking via Context-aware Auto-prompting and Instruction-following Contrastive Decoding (2024)

Abstract: Zero-shot cross-domain dialogue state tracking (DST) enables us to manage task-oriented dialogues in new, unseen domains without the cost of collecting in-domain data. Previous studies have implemented slot-based input improvements, such as schema-driven descriptions and question-answering formats, but still suffer from negative transfer for seen slots and inefficient transfer for unseen slots due to the significant source-target domain gap. To address these issues, we introduce a novel framework called Context-aware Auto-prompting and Instruction-following Contrastive Decoding (CAPID). This framework generates dynamic, context-aware slot queries, effectively improving the model’s transferability. Our context-aware auto-prompting approach tailors slot queries to the current dialogue context, increasing flexibility and reducing ambiguities. Additionally, an instruction-following contrastive decoding strategy helps reduce errors related to off-topic slots by penalizing deviations from the provided instructions. Extensive experiments on two datasets, with varying model sizes (from 60M to 7B), demonstrate the superior performance of CAPID. The source code is provided for reproducibility.

## introduction task-oriented dialogue systems are powerful tools for assisting users in accomplishing a wide range of tasks, e.g. booking a train or making restaurant reservations @xcite . dialogue state tracking (dst), a crucial component of these systems, is responsible for identifying domain-slot pairs (e.g., <restaurant-@xmath0.g., "eraina") from the conversation @xcite . in industrial applications, these systems frequently need to incorporate new domains based well what hotel will you be staying at? hello! what can i do for you? figure 1 : an example illustrating negative transfer in zero-shot cross-domain dst. after training on source domains-hotel, taxi, and others-the model is tested on the restaurant domain to evaluate its cross-domain transfer ability. previous methods using static slot descriptions erroneously assign the value "bridge guest house," which belongs to the previously trained "hotel" domain, to @xmath1. conversely, our context-aware slot query approach allows the model to accurately identify "eraina" as the correct value. on user requirements. however, collecting extensive data for each new domain is both costly and inefficient. consequently, there is a substantial need for zero-shot cross-domain dst, which requires a dst model to adapt to entirely new target domains without retraining, leveraging knowledge already acquired in previous source domains @xcite . this transferability is essential for developing flexible and scalable dialogue systems that can operate effectively across diverse domains without exhaustive, domain-specific training. with the impressive performance demonstrated by large-scale pretrained neural language models (lms), recent studies have focused on using slotrelated descriptions as input to enhance slot understanding @xcite @xcite . for example, in order to mitigate the confusion arising from abbreviated slots (e.g., "ar-riveby," which does not clearly indicate whether it refers to a time or a method of arrival), @xcite used schema-driven prompting by incorporating natural language schema descriptions, resulting in remarkable performance improvement. additionally, @xcite and @xcite proposed to convert conventional slot descriptions into question-answering formats, further enhancing clarity and understanding. building on these recent advancements, our investigation of current works on zero-shot cross-domain dst (see section 2 for an in-depth analysis) has uncovered the following previously unreported challenge: challenge 1: negative transfer for seen slots @xcite . this occurs when a slot in the target domain has already been encountered in the previously trained source domain. as illustrated in figure 1 , @xmath2. when tracking its value in the target "restaurant" domain, previous methods erroneously apply learned patterns from previously trained domains, resulting in incorrect responses. challenge 2: inefficient transfer for unseen slots. this arises when the model encounters a slot that has never been seen in the source domain. due to the significant domain gap, the model struggles to transfer existing knowledge effectively. these challenges arise because of the fixed descriptions for each slot used by traditional methods, which are collected by crowd-sourced human annotators and might be inconsistent across different domains. moreover, these slot-based enhancements lead the model to learn fixed slot-value pairs, lacking the contextual diversity and nuanced understanding achieved from a broader contextual perspective. for instance, contexts such as "expensive" and "european" in figure 1 would help the model better differentiate between domains. to address these challenges, we propose contextaware auto-prompting and instruction-following contrastive decoding (capid), a context-aware slot query generation approach through autoprompting to align the gap between source and target domains. rather than generating prompts solely related to slots, our method is the first to dynamically creates customized prompts for each sample by also incorporating contextual informa-tion from the dialogue. the detailed comparisons between our method and the previous dst methods can be shown in we utilize auto-prompting capability of gpt-4 @xcite to generate contextaware slots by incorporating dialogue contexts and domain-specific attributes. what's more, when training with these context-aware slots changing dynamically with the dialogue, the model accumulate the knowledge of analyzing dialogue contexts to extract slot values, which is more tranferrable to unseen slots. while gpt-4 demonstrates remarkable capabilities, its usage limits and data privacy issues necessitate a different strategy for large-scale slot query generation. we address this by training a smaller student model to emulate gpt-4's contextaware slot query generation. furthermore, to ensure the student model adheres to instructions when handling off-topic slots, we developed a novel instruction-following contrastive decoding method. this method maximizes the difference between expert log-probabilities from the original prompt and amateur log-probabilities from the repurposed prompt, restricting the search space to high-probability tokens that are semantically consistent with the original prompt, thereby producing coherent results. to summarize, our main contributions include: ## motivation we categorize slots into two types: unseen slots, which lack counterparts in the source domains (e.g., "<@xmath3. "<@xmath4, and seen slots, which exhibit similarities across domains (e.g., "<@xmath5. ## conclusion we have introduced capid, a novel framework that creates dynamic, context-aware slot prompts for zero-shot cross-domain dst. by utilizing these context-aware slot prompts, we have substantially reduced the complexity of dst, enhancing the model's comprehension and adaptability to new domains. the combination of a student model trained on context-aware slot prompts with our instructionfollowing contrastive decoding method has significantly reduced errors on off-topic slots. extensive experiments on two datasets demonstrate the superior performance of the capid framework, highlighting its robustness and versatility across various language models.