C ross WOZ : A Large-Scale C hinese Cross-Domain Task-Oriented Dialogue Dataset (2020)

Abstract: To advance multi-domain (cross-domain) dialogue modeling as well as alleviate the shortage of Chinese task-oriented datasets, we propose CrossWOZ, the first large-scale Chinese Cross-Domain Wizard-of-Oz task-oriented dataset. It contains 6K dialogue sessions and 102K utterances for 5 domains, including hotel, restaurant, attraction, metro, and taxi. Moreover, the corpus contains rich annotation of dialogue states and dialogue acts on both user and system sides. About 60% of the dialogues have cross-domain user goals that favor inter-domain dependency and encourage natural transition across domains in conversation. We also provide a user simulator and several benchmark models for pipelined task-oriented dialogue systems, which will facilitate researchers to compare and evaluate their models on this corpus. The large size and rich annotation of CrossWOZ make it suitable to investigate a variety of tasks in cross-domain dialogue modeling, such as dialogue state tracking, policy learning, user simulation, etc.

## introduction recently, there have been a variety of task-oriented dialogue models thanks to the prosperity of neural architectures @xcite @xcite @xcite . however, research is still largely limited by the lack of large-scale highquality dialogue data. many corpora have advanced the research of task-oriented dialogue systems, most of which are single domain conversations, including atis @xcite , dstc 2 @xcite , frames @xcite , kvret @xcite , woz 2.0 @xcite , and m2m @xcite . ## related work according to whether the dialogue agent is human or machine, we can group the collection methods of existing task-oriented dialogue datasets into three categories. the first one is human-to-human dialogues. one of the earliest and well-known is the atis dataset @xcite an attraction and one of its nearby hotels, then booking a taxi to commute between these two places. in addition to expressing prespecified informable slots and filling in requestable slots, users need to consider and modify cross-domain informable slots (bold) that vary through conversation. we only show a few turns (turn number on the left), each with either user or system state of the current domain, which are shown above each utterance. @xcite . the performance of the dialogue system will largely influence the quality of dialogue data. the third one is machine-tomachine dialogues. it needs to build both user and multiwoz crosswoz usr: i'm looking for a college type attraction. usr: i like beijing happy valley. what hotels are around this attraction? schema sys: usr: i want a hotel in san diego and i want to there are many, such as hotel a, hotel b, and hotel c. check out on thursday next week. usr: . . . usr: i need a one way flight to go there. great! i am planning to find a hotel to stay near the attraction. which one has a rating of 4 or higher and offers wake-up call service? it needs much less human effort. however, the complexity and diversity of dialogue policy are limited by the simulators. to explore dialogue policy in multi-domain scenarios, and to collect natural and diverse dialogues, we resort to the human-to-human setting. most of the existing datasets only involve single domain in one dialogue, except multiwoz @xcite and schema @xcite . the multiwoz dataset has attracted much attention recently, due to its large size and multi-domain characteristics. it is at least one order of magnitude larger than previous datasets, amounting to 8,438 dialogues and 115k turns in the training set. it greatly promotes the research on multi-domain dialogue modeling, such as policy learning @xcite , state tracking @xcite , and context-to-text generation @xcite . recently the schema dataset has been collected in a machine-to-machine fashion, resulting in 16,142 dialogues and 330k turns for 16 domains in the training set. however, the multi-domain dependency in these two datasets is only embodied in imposing the same pre-specified constraints on different domains, such as requiring a restaurant and an attraction to locate in the same area, or the city of a hotel and the destination of a flight to be the same ( to better track the conversation flow and model user dialogue policy, we provide annotation of user states in addition to system states and dialogue acts. while the system state tracks the dialogue history, the user state is maintained by the user and indicates whether the sub-goals have been completed, which can be used to predict user actions. this information will facilitate the construction of the user simulator. to the best of our knowledge, crosswoz is the first large-scale chinese dataset for task-oriented dialogue systems, which will largely alleviate the shortage of chinese task-oriented dialogue corpora that are publicly available. ## data collection our corpus is to simulate scenarios where a traveler seeks tourism information and plans her or his travel in beijing. domains include hotel, attraction, restaurant, metro, and taxi. the data collection process is summarized as follows: ## statistics after removing uncompleted dialogues, we collected 6,012 dialogues in total. the dataset is split randomly for training/validation/test, where the statistics are shown in according to the type of user goal, we group the dialogues in the training set into five categories: single-domain (s) 417 dialogues have only one sub-goal in har domains. independent multi-domain (m)1,573 dialogues have multiple sub-goals (2∼3) in har domains. however, these sub-goals do not have cross-domain informable slots. the data statistics are shown in cm and cm+t are much more challenging than other tasks because additional cross-domain constraints in har domains are strict and will result in more ''nooffer'' situations (i.e., the wizard finds no result that satisfies the current constraints). in this situation, the wizard will try to relax some constraints and issue multiple queries to find some results for a recommendation while the user will compromise and change the original goal. the negotiation process is captured by ''nooffer rate'', ''multi-query rate'', and ''goal change rate'' in the distribution of dialogue length is shown in figure 2 , which is an indicator of the task complexity. most single-domain dialogues terminate within 10 turns. the curves of m and m+t are almost of the same shape, which implies that the traffic task requires two additional turns on average to complete the task. the curves of cm and cm+t are less similar. this is probably because cm goals that have 5 sub-goals (about 22%) can not further generate a sub-goal in traffic domains and become cm+t goals. ## corpus features our corpus is unique in the following aspects: ## benchmark and analysis crosswoz can be used in different tasks or settings of a task-oriented dialogue system. to facilitate further research, we provide benchmark models for different components of a pipelined task-oriented dialogue system (figure 3 ), including natural language understanding (nlu), dialogue state tracking (dst), dialogue policy learning, and natural language generation (nlg). these models are implemented using convlab-2 @xcite , an open-source task-oriented dialog system toolkit. we also provide a rulebased user simulator, which can be used to train dialogue policy and generate simulated dialogue data. the benchmark models and simulator will greatly facilitate researchers to compare and evaluate their models on our corpus. ## conclusion in this paper, we present the first large-scale chinese cross-domain task-oriented dialogue dataset, crosswoz. it contains 6k dialogues and 102k utterances for 5 domains, with the annotation of dialogue states and dialogue acts at both user and system sides. about 60% of the dialogues have cross-domain user goals, which encourage natural transition between related domains. thanks to the rich annotation of dialogue states and dialogue acts at both user side and system side, this corpus provides a new testbed for a wide range of tasks to investigate crossdomain dialogue modeling, such as dialogue state tracking, policy learning, and so forth. our experiments show that the cross-domain constraints are challenging for all these tasks. the transition between related domains is especially challenging to model. besides corpus-based component-wise evaluation, we also performed system-level evaluation with a user simulator, which requires more powerful models for all components of a pipelined cross-domain dialogue system.