N eural WOZ : Learning to Collect Task-Oriented Dialogue via Model-Based Simulation (2021)

Abstract: We propose NeuralWOZ, a novel dialogue collection framework that uses model-based dialogue simulation. NeuralWOZ has two pipelined models, Collector and Labeler. Collector generates dialogues from (1) user’s goal instructions, which are the user context and task constraints in natural language, and (2) system’s API call results, which is a list of possible query responses for user requests from the given knowledge base. Labeler annotates the generated dialogue by formulating the annotation as a multiple-choice problem, in which the candidate labels are extracted from goal instructions and API call results. We demonstrate the effectiveness of the proposed method in the zero-shot domain transfer learning for dialogue state tracking. In the evaluation, the synthetic dialogue corpus generated from NeuralWOZ achieves a new state-of-the-art with improvements of 4.4% point joint goal accuracy on average across domains, and improvements of 5.7% point of zero-shot coverage against the MultiWOZ 2.1 dataset.

## introduction for a task-oriented dialogue system to be scalable, the dialogue system needs to be able to quickly adapt and expand to new scenarios and domains. however, the cost and effort in collecting and annotating an expanding dataset is not only laborintensive but also proportional to the size and variety of the unseen scenarios. there are three types of dialogue system expansions. (1) the simplest expansion is the addition of new instances in the knowledge base (kb) under the identical schema. for example, the addition of newly opened restaurants in the kb of restaurant domain falls under this category. (2) a slightly more complicated expansion involves modifications to the kb schema, and possibly the related 1 the code is available at github.com/naver-ai/neuralwoz. figure 1 : overview of neuralwoz. the neuralwoz takes goal instruction for the user side (u) and api call results for the system side (s) to synthesize dialogue. first, it generates dialogue from the inputs and then labels dialogue state (b t ) and active domain (domain t ) by turn t on the dialogue. to support development of scalable dialogue systems, we propose neuralwoz, a model-based dialogue collection framework. neuralwoz uses goal instructions and kb instances for synthetic dialogue generation. neuralwoz mimics the mechanism of a wizard-of-oz @xcite and figure 1 illustrates our approach. neuralwoz has two neural components, collector and labeler. collector generates a dialogue by using the given goal instruction and candidate relevant api call results from the kb as an input. labeler annotates the generated dialogue with appropriate labels by using the schema structure of the dialogue domain as meta information. more specifically, labeler selects the labels from candidate labels which can be obtained from the goal instruction and the api call results. as a result, neuralwoz is able to generate a dialogue corpus without training data of the target domain. we evaluate our method for zero-shot domain transfer task @xcite to demonstrate the ability to generate corpus for unseen domains, when no prior training data exists. in dialogue state tracking (dst) task with multiwoz 2.1 @xcite , the synthetic data generated with neuralwoz achieves 4.4% point higher joint goal accuracy and 5.7% point higher zero-shot coverage than the existing baseline. additionally, we examine few-shot and full data augmentation tasks using both training data and synthetic data. we also illustrate how to collect synthetic data beyond multiwoz domains, and discuss the effectiveness of the proposed approach as a data collection strategy. our contributions are as follows: ## neuralwoz in this section, we describe the components of neu-ralwoz in detail, and how they interact with each other. figure 2 illustrates the input and output of two modules in neuralwoz. the synthetic corpus, which collector and labeler made, are used for the training of the dst baselines, trade @xcite and sumbt @xcite in our experiments. ## experimental results we use both joint goal accuracy (jga) and slot accuracy (sa) as the performance measurement. the jga is an accuracy which checks whether all slot values predicted at each turn exactly match the ground truth values, and the sa is the slotwise accuracy of partial match against the grouth truth values. especially for zero and few-shot setting, we follow the previous setup @xcite . following campagna et al. ( 2020 ), the zero-shot learner model should be trained on data excluding the target domain, and tested on the target domain. we also add synthesized data from our neuralwoz which is trained in the same way, i.e., leave-one-out setup, to the training data in the experiment. ## analysis 6.1 error analysis figure 3 shows the slot accuracy for each slot type in the hotel domain, which is the weakest domain from ours. different from other four domains, only the hotel domain has two boolean type slots, "parking" and "internet", which can have only "yes" or "no" as their value. since they have abstract property for the tracking, labeler's labeling performance tends to be limited to this domain. however, it is noticeable that our accuracy of booking related slots (book stay, book people, book day) are much higher than the atdm's. moreover, the model using synthetic data from the atdm totally fails to track the "book stay" slot. in the synthesizing procedures of @xcite , they create the data with a simple substitution of a domain noun phrase when the two domains have similar slots. for example, "find me a restaurant in the city center" can be replaced with "find me a hotel in the city center" since the restaurant and hotel domains share "area" slot. we presume it is why they outperform over slots like "pricerange" and "area". ## conclusion we propose neuralwoz, a novel dialogue collection framework, and we show our method achieves state-of-the-art performance on zero-shot domain transfer task. we find the dialogue corpus from neuralwoz is synergetic with human-annotated data. finally, further analysis shows that neural-woz can be applied for scaling dialogue system. we believe neuralwoz will spark further research into dialogue system environments where expansion target domains are distant from the source domains.