R i SAWOZ : A Large-Scale Multi-Domain W izard-of- O z Dataset with Rich Semantic Annotations for Task-Oriented Dialogue Modeling (2020)

Abstract: In order to alleviate the shortage of multi-domain data and to capture discourse phenomena for task-oriented dialogue modeling, we propose RiSAWOZ, a large-scale multi-domain Chinese Wizard-of-Oz dataset with Rich Semantic Annotations. RiSAWOZ contains 11.2K human-to-human (H2H) multi-turn semantically annotated dialogues, with more than 150K utterances spanning over 12 domains, which is larger than all previous annotated H2H conversational datasets. Both single- and multi-domain dialogues are constructed, accounting for 65% and 35%, respectively. Each dialogue is labeled with comprehensive dialogue annotations, including dialogue goal in the form of natural language description, domain, dialogue states and acts at both the user and system side. In addition to traditional dialogue annotations, we especially provide linguistic annotations on discourse phenomena, e.g., ellipsis and coreference, in dialogues, which are useful for dialogue coreference and ellipsis resolution tasks. Apart from the fully annotated dataset, we also present a detailed description of the data collection procedure, statistics and analysis of the dataset. A series of benchmark models and results are reported, including natural language understanding (intent detection & slot filling), dialogue state tracking and dialogue context-to-text generation, as well as coreference and ellipsis resolution, which facilitate the baseline comparison for future research on this corpus.

## introduction in recent years, we have witnessed that a variety of datasets tailored for task-oriented dialogue have been constructed, such as multiwoz @xcite , sgd @xcite and @xcite , along with the increasing interest in conversational ai in both academia and industry @xcite . these datasets have triggered extensive research in either end-to-end or traditional modular taskoriented dialogue modeling @xcite . despite of substantial progress made based on these newly built corpora, more efforts in creating challenging datasets in terms of size, multiple domains, semantic annotations, multilinguality, etc., are still in demand @xcite . among the existing datasets, the majority of them are not large in size, e.g., atis @xcite , woz 2.0 @xcite , frames @xcite and kvret @xcite , which might not well support datahungry neural dialogue models. very large taskoriented dialogue datasets can be created in a machine-to-machine fashion, such as m2m @xcite and sgd @xcite . datasets collected in this way need to simulate both user and system and contain unnatural conversations. multiwoz @xcite , probably the most promising and notable dialogue corpus collected in a wizard-of-oz (i.e., human-to-human) way recently, is one order of magnitude larger than the aforementioned corpora collected in the same way. however, it contains noisy systemside state annotations and lacks user-side dialogue acts foot_1 @xcite . yet another very recent dataset @xcite , the first large-scale chinese h2h dataset for task-oriented dialogue, provides semantic annotations on both user and system side although it is relatively smaller than multiwoz. the number of domains in both multiwoz and crosswoz is it contains 11,200 multi-turn dialogues with more than 150k utterances ranging over 12 domains, namely attraction, restaurant, hotel, flight, train, weather, movie, tv, computer, car, hospital and education (particularly after-school remedial courses), several of which are not covered in previous datasets. compared with other wizard-of-oz datasets (e.g. multiwoz and crosswoz), risawoz offers a wider and more balanced coverage on both domains and slots, making it suitable for not only practical use in industrial scenarios of related domains but also research on domain adaptation, few/zero-shot learning in task-oriented dia-user goal: 你刚到苏州旅游，想咨询些信息。首先，你想在姑苏区找一家价格便宜的酒店，询问酒 店的房费和地址。其次，你希望客服推荐这附近最热门的苏州园林，并询问它的地址和 特点。然后，你还想在附近找一家有名的江浙菜餐厅，并询问他们家的推荐菜。最后你 感谢客服的帮助并说再见。 you have just arrived in suzhou for a tour and want to ask for some information. first of all, you want to find a cheap hotel in gusu district, and ask about the hotel's room rate and address. secondly, you would like the customer service to recommend the most popular suzhou garden nearby, and ask about its address and characteristics. then, you want to find a famous restaurant nearby which serves jiangsu and zhejiang cuisine and ask about recommended dishes of the restaurant. finally, you thank the customer service for his help and say goodbye. dialogue domains: [hotel, attraction, restaurant] 你好。我是来苏州旅游的，我能咨询一些姑苏区的酒店信息吗？ hello. i'm traveling in suzhou. can i ask for some information about hotels in gusu district? user actions: general: greeting, hotel: inform (@xmath0: hotel: @xmath1, glad to serve you. there are many hotels in that area. what price range do you prefer? system actions: general: greeting, hotel: request (@xmath2. how much is the room rate? user actions: hotel: inform (@xmath3, hotel: request (room @xmath4: hotel: @xmath5, @xmath6. how much is the room rate of the hotel? rewriting utterances for ellipsis and coreference resolution 我们为您推荐速8酒店，他们家的标准间是238元每晚。 we recommend super 8 hotel for you. their standard room is 238 yuan per night. system actions: hotel: recommend (@xmath7, inform (room @xmath8. . . co-reference clusters: cluster 1 : [姑苏区 (gusu district), 那里 (that area), 那儿 (there), . . . ] cluster 2 : [速8酒店 (super 8 hotel), 他们家 (their), . . . ] ## related work we follow @xcite to roughly categorize existing task-oriented dialogue datasets into three groups: machine-to-machine, humanto-machine, and human-to-human. from the perspective of domain quantity and data scale, most existing datasets cover only one single or a few domains while large-scale multi-domain datasets are not widely available. as suggested by @xcite , task-oriented dialogue datasets in languages other than english are few. to the best of our knowledge, there has been no large-scale dialogue datasets with linguistic annotations aiming at ubiquitous discourse phenomena (e.g., ellipsis and coreference) in dialogue. although some recent works have proposed datasets with utterance completion annotation for ellipsis or coreference in dialogue @xcite , these datasets are at small scale and with simple dialogue goals. no dialogue datasets provide annotations of coreference clusters. machine-to-machine collecting data of this type requires to create a user and system simulator to generate multi-turn dialogue outlines, which are further transformed into natural language utterances via paraphrasing with predefined rules or crowdsourced workers @xcite . despite of less human effort required in this approach, the diversity and complexity of created dialogues greatly depend on the quality of user and system simulators. it's also difficult to avoid mismatch between machine-created dialogues and real human conversations. human-to-machine in this method, humans converse with an existing dialogue system to collect dialogue data. the dialogue state tracking challenges (dstc) has provided several datasets created in this way @xcite . generally, the quality of human-to-machine data heavily relies on the performance of the given dialogue system. human-to-human to collect data of this type, crowdsourced workers talk to each other according to given dialogue goals to create diverse and natural dialogues. atis @xcite , woz 2.0 @xcite , frames @xcite and kvret @xcite are small-scale datasets built in this way. in contrast, @xcite and cross-woz @xcite are two large-scale h2h datasets proposed recently. coreference resolution coreference is ubiquitous in dialogue. however, there is no available dialogue dataset with labeled coreference clusters. generally, coreference datasets are created on text paragraphs or documents. the ontonotes 5.0 dataset @xcite is one of the most widely-used document-level dataset on coreference resolution from the conll-2012 shared task. ## dataset creation the whole process of data collection consists of database and ontology construction, goal generation, dialogue collection and two rounds of annotations. ## our dialogue dataset our dataset contains not only single-domain dialogues, but also a great amount of multi-domain dialogues where domains are naturally connected. for example, a user wants to travel from one place to another. after checking the air ticket or train ticket, she wants to ask for the local weather information as well. in this section, we will introduce our dataset from two aspects: data structure and data statistics. ## risawoz as a new benchmark the large size and rich semantic annotations of risawoz make it a suitable testbed for various benchmark tasks. we conduct five different evaluation tasks with the benchmark models and in-depth analyses on risawoz in this section. we also discuss the applicability of risawoz for other tasks. results of the 5 tasks are reported in ## conclusion in this paper, we have presented risawoz, to date the largest human-to-human multi-domain dataset annotated with rich semantic information for taskoriented dialogue modeling. we manually label each dialogue in risawoz not only with comprehensive dialogue annotations for various sub-tasks of task-oriented dialogue systems (e.g., nlu, dst, response generation), but also linguistic annotations over ellipsis and coreference in multi-turn dialogue. in addition, the process of data creation and annotation is described in detail. we also report the benchmark models and results of five evaluation tasks on risawoz, indicating that the dataset is a challenging testbed for future work. risawoz is featured with large scale, wide domain coverage, rich semantic annotation and functional diversity, which can facilitate the research of task-oriented dialogue modeling from different aspects.