CMCC : A Comprehensive and Large-Scale Human-Human Dataset for Dialogue Systems (2022)

Abstract: Dialogue modeling problems severely limit the real-world deployment of neural conversational models and building a human-like dialogue agent is an extremely challenging task. Recently, data-driven models become more and more prevalent which need a huge amount of conversation data. In this paper, we release around 100,000 dialogue, which come from real-world dialogue transcripts between real users and customer-service staffs. We call this dataset as CMCC (China Mobile Customer Care) dataset, which differs from existing dialogue datasets in both size and nature significantly. The dataset reflects several characteristics of human-human conversations, e.g., task-driven, care-oriented, and long-term dependency among the context. It also covers various dialogue types including task-oriented, chitchat and conversational recommendation in real-world scenarios. To our knowledge, CMCC is the largest real human-human spoken dialogue dataset and has dozens of times the data scale of others, which shall significantly promote the training and evaluation of dialogue modeling methods. The results of extensive experiments indicate that CMCC is challenging and needs further effort. We hope that this resource will allow for more effective models across various dialogue sub-problems to be built in the future.

## introduction task-oriented dialogue systems @xcite @xcite @xcite are designed to assist user in completing daily tasks, which involve reasoning over multiple dialogue turns. tremendous progress has been made recently, but building a human-like dialogue system is a challenging task remaining. to drive the progress of building dialogue systems using data-driven approaches, a number of conversational corpora have been released in the past. task-oriented dialogue corpus, such as frames @xcite , multiwoz @xcite , crosswoz @xcite , risawoz @xcite , are collected by two crowd workers playing the roles of the user and the system, which often leads to be smallscale, and can not sufficiently capture a number of challenges that arise with production scaling. more recently, some researchers construct dialogue datasets from real human-to-human scenario conversations, especially human-to-human customer service scenario, such as jddc @xcite and mobilecs @xcite . jddc is collected from e-commerce scenario and annotates intent information. mobilecs is conducted from mobile customer service scenario and model the process as task-oriented conversations. therefore, the entity information related to tasks is annotated. however, the complexity of the dialogue process is far more than tod, in addition to task completion, it is also accompanied by emotional support that appease an angry customer and providing solutions. several emotional support conversation corpora @xcite @xcite are designed to emotional chat or provide empathetic responding. since the emotional supporters are not welltrained, existing datasets do not naturally exhibit examples or elements of supportive conversations. as a result, data-driven models which leverage such corpora are limited in their ability to explicitly learn how to provide effective support. esconv @xcite is collected by communication of trained individuals who play the roles of the seeker and the supporter, and guided by predefined emotional support conversation framework, however, it is more focused on alleviating the negative emotions that users encounter in their daily lives. despite the efforts in modeling emotional support, work that focuses specifically on modeling emotional care and support in task-oriented dia-logue system is relatively limited. to this end, we design a customer service care-oriented taxonomy, and annotate care-oriented information for mo-bilecs dataset, covering 9 types of emotion labels and 17 types of customer service act labels finally. this new dataset consists of two parts, 8975 dialogues which are labeled with annotations of careoriented information and other more than 90,000 unlabeled dialogues. we call this new dataset as cmcc (china mobile customer care) dataset. to be able to explain the patterns and trends of the conversation flow, we employ visualization methods to illustrate the most frequent exchanges and reveal how they temporally vary as dialogues proceed. finally, we explore and demonstrate the effectiveness of care-oriented information for dialogue sub-tasks. we highlight our contributions as follows: ## data characteristics this section mainly introduces the characteristics of the data. in addition to showing the number of conversations and labels in the dataset, we also demonstrate the characteristics of customer service dialogue data by visualizing the transition between customer service acts and user emotion in dialogues. ## experiments in this section, we conduct experiments on the cmcc dataset. we focus on two tasks: dialogue response generation and user emotion recognition. ## conclusion in this paper, we present cmcc, to date the largest human-to-human real-life dataset annotated with rich care-oriented information on top of mobilecs. we not only manually label each dialogue with comprehensive user emotion, customer service act and satisfaction annotations for various sub-tasks of multi-domain dialogue systems, but also further investigate approach to facilitate the research of care-oriented way via empirical experiments. in addition, the process of data annotation and visualization is described in detail. we also report the benchmark models and results of two evaluation tasks on cmcc, indicating that the dataset is a challenging testbed for future work. we will enrich the dataset annotations (e.g., solutions, external knowledge and api calls) from various aspects in future work. we hope it can bring more imagination and benefit future research in dialogue systems.