UNO - DST : Leveraging Unlabelled Data in Zero-Shot Dialogue State Tracking (2024)

Abstract: Previous zero-shot dialogue state tracking (DST) methods only apply transfer learning, but ignore unlabelled data in the target domain.We transform zero-shot DST into few-shot DST by utilising such unlabelled data via joint and self-training methods. Our method incorporates auxiliary tasks that generate slot types as inverse prompts for main tasks, creating slot values during joint training. Cycle consistency between these two tasks enables the generation and selection of quality samples in unknown target domains for subsequent fine-tuning. This approach also facilitates automatic label creation, thereby optimizing the training and fine-tuning of DST models. We demonstrate this method’s effectiveness on general language models in zero-shot scenarios, improving average joint goal accuracy by 8% across all domains in MultiWOZ.

## introduction dialogue state tracking (dst) is a crucial task in understanding users' intentions by extracting the dialogue states from the dialogue history @xcite , where a single dialogue state is a pairing of a slot type (e.g.,<hotel-@xmath0.g.,<hilton @xmath1, as in figure 1 . dialogue states are a set of those combinations (e.g.,<hotel-name: hilton @xmath2, given dialogue history and slot types. traditional methods train and evaluate dst models with manually-labelled dialogue states in each domain, which can be costly and time-consuming @xcite . recently, dst under zero and few-shot settings draw increased attention @xcite . compared with few-shot methods, zero-shot approaches are more challenging, due to unseen slot types and data scarcity in unknown target domains. in both zero and few-shot settings, the majority of existing methods convert the dst problem into other common problem settings in natural language processing (nlp): for example, question answering (qa; @xcite , prompt learning @xcite , summarization @xcite and instruction learning @xcite . for example in figure 1 , a given slot type (<hotel-@xmath3. transfer learning methods also convert dst tasks to generation ones, more suited for pretrained language models (lms; @xcite . however, such methods cannot fully leverage the capability of lms in generation and selection. two main difficulties emerge: 1) the performance of the chosen nlp tasks can be unpredictable for unseen slot types in a new domain due to domain divergence; and 2) existing models are only trained in the known domains, without utilizing any unlabeled data in the new target domain. this work proposes uno-dst 2 , a method to leverage the unlabelled data for zero-shot dst in target domain. inspired by the popularity of multi-task learning and self-supervised learning @xcite , uno-dst employs a two-step training framework invoking both joint and self-training (figure 3 ). aside from the main task of generating slot values, we design an auxiliary task of generating slot types. we then jointly train both tasks using the labelled training data in source domains. for the self-training period, we implement the concept of cycle consistency within our two tasks @xcite . that is, a text output from the main task serves as input to the auxiliary task, and the resultant text produced by the auxiliary task should match the original input text (figure 2 ). this process forms a full cycle, ensuring consistent generation and selection of dialogue states from the unlabelled data, which is further used for fine-tuning the model. in this way, we convert zero-shot problems into few-shot ones. importantly, our framework is model-agnostic which applies to different baseline models. our main contributions are as follows: ## related works existing dst methods are generally classified as either 1) full-data or 2) low-resource dst. despite the method chosen, unlabeled training data in the target domain remains unexploited; in fewshot dst, although pseudo labels can be derived from unlabeled data in the same domain @xcite , there is a notable absence of research on target domain unlabelled data in zero-shot dst. full-data dst are commonly trained with fully annotated multi-domain conversations @xcite . sota models focus on dst tasks with well-annotated datasets @xcite . however, the annotation work for data in a new domain can be costly. hence there is interest in transferring the knowledge of a model from a known domain into an unknown domain and conducting dst tasks in a low-resource setting. low-resource dst uses zero-or few-shot learning in the unknown target domain. here, the state-of-the-art use a single nlp task to transfer knowledge from the source domains to the unknown target domain @xcite . while transfer learning tasks achieve good results, each method is task-dependent. thus, taskindependent strategies have been proposed @xcite . multi-task learning involves simultaneous training of a model on diverse tasks, to boost performance on trained downstream tasks. it also holds promise for enhancements on new tasks @xcite . however, existing methods typically neglect to assess the consistency across multiple tasks after joint training, while our approach leverages the cycle consistency for selection @xcite . ## methodology in figure 3 , we show an overview of uno-dst with joint training and self-training periods. our method includes two tasks: a main task for slot value prediction ( §3.1) and an auxiliary task for slot type prediction ( §3.2). in the joint training period, both tasks are jointly trained in the known source domains ( §3.3). in the self-training period, we introduce three steps to generate dialogue states, select good samples, and fine-tune the lm ( §3.4). lastly, we elaborate on the transferability of our strategy with an oracular selection approach ( §3.5). ## joint training with auxiliary tasks we conduct a simple version of joint training with only two tasks: the main and the auxiliary dst tasks. the training samples for the main tasks are created using dialogue history and slot type, while the samples for the auxiliary dst tasks are created by masking the slot values from dialogue history. as the auxiliary task is an inverse process of the main task, the model is trained for the same knowledge in a cycle-consistent way. by predicting the masked slot type from the masked dialogue history, the model is familiar with the context and different slot types. with our specially designed auxiliary task, the generation model reuses the existing data for another round of training without the need to increase the amount of training data or model parameters. we formulate the loss function for the main task l m and auxiliary task l a as: the final loss is a simple average of both. to keep the process simple, we do not add hyperparameters to the model framework. importantly, as the auxiliary task samples are generated using the inverse prompt of the main task, the ratio of these two tasks mirrors the natural distribution of both tasks throughout the joint training period. ## experiments and datasets dataset. we train and test our model on both multiwoz 2.1 @xcite and the schema-guided dialogue (sgd; @xcite . multiwoz and sgd have dialogues distributed in both training and testing distributions evaluation metrics. the primary metric for dst evaluation is joint goal accuracy (jga), which compares the set of generated predicted values with the set of ground truth ones after each conversation turn and average goal accuracy (aga) calculates the jga only for active slot types as in sgd dataset @xcite . baselines and experiment setup. we use t5 @xcite as our baseline model. for a fair analysis, we also compare our results with previous dst benchmarks: trade @xcite and the sgd baseline @xcite , and current sota models: t5dst @xcite , transferqa @xcite and sd-t5 @xcite . we adopt the cross-domain settings @xcite for both datasets, experimenting on two checkpoints, "t5-small" foot_1 and "t5-qa" foot_2 . the unlabelled training data in the target domain will be used for self-training and testing data in target domain is only used for final testing. we select qa as the main task in our framework for its popularity and test it through different checkpoints holding parameters fixed. we adopt the open-source "t5-small" @xcite with 60m parameters as our baseline and train using adamw with a learning rate of 0.0001, batch size 8 for 1 epoch (zero-shot setting) and 3 epochs (fine-tuning setting) on single geforce rtx3090. ## results ## uno-dst with chatgpt while earlier sections analyse the effectiveness of our methods on lms like "t5", this section focuses on the potential application of uno-dst with large language models (llms), such as @xcite . specifically, we examine uno-dst for zero-shot dst using openai's chatgpt foot_3 as the backbone llm, an llm which has been adopted as a language tool for information extraction with strong capabilities, even without specific training or fine-tuning. our study will test the efficacy of our selftraining strategy in uno-dst on chatgpt, includ- all generated slot types in train domain people 5 , day 5 , destination 5 , departure 5 , leave 5 , arrive 5 , price 5 , type 5 , time 5 , area 5 , name 5 valid new slot types price 15 , day 1 , parking 3 , name 5 dialogue example [pmul1359] (price 5 , name 5 ) system:"okay, tr6572 departs at 05:29." user:"what is the price?" dialogue example [pmul3027] (parking 3 ) system:"i have 2 turkish restaurants in the centre?" user:"do they offer free parking?" dialogue example [pmul1118] (day 1 ) user:"i am in cambridge for the week and want to know what museums you guys have there." ing the web interface for conversational approach ( §7.1) and the api 6 for larger testing corpus using in-context learning (icl; @xcite ; §7.2). ## conclusion we propose a novel approach to convert the zeroshot dst into a few-shot setting by generating and selecting quality dialogue states from unlabeled data in the target domain through joint and self-training periods. we introduce and demonstrate how our proposed auxiliary task, which generates slot types as the inverse prompt for the main task which generates slot values, serves the whole model for 1) better accuracy of the main task in joint training 2) quality data selection in the selftraining period 3) new slot types generation beyond the given slot type list and 4) upgrading to llms. our proposed strategies of uno-dst are taskindependent, which can be extended to other prompt formats and generalised to llms. we look forward to future works that engage additional auxiliary tasks which target new datasets and apply zero-shot dst, even where no slot types are given.