CG o D ial: A Large-Scale Benchmark for C hinese Goal-oriented Dialog Evaluation (2022)

Abstract: Practical dialog systems need to deal with various knowledge sources, noisy user expressions, and the shortage of annotated data. To better solve the above problems, we propose CGoDial, a new challenging and comprehensive Chinese benchmark for multi-domain Goal-oriented Dialog evaluation. It contains 96,763 dialog sessions, and 574,949 dialog turns totally, covering three datasets with different knowledge sources: 1) a slot-based dialog (SBD) dataset with table-formed knowledge, 2) a flow-based dialog (FBD) dataset with tree-formed knowledge, and a retrieval-based dialog (RBD) dataset with candidate-formed knowledge. To bridge the gap between academic benchmarks and spoken dialog scenarios, we either collect data from real conversations or add spoken features to existing datasets via crowd-sourcing. The proposed experimental settings include the combinations of training with either the entire training set or a few-shot training set, and testing with either the standard test set or a hard test subset, which can assess model capabilities in terms of general prediction, fast adaptability and reliable robustness.

## introduction goal-oriented dialog systems converse with users naturally, helping them fulfill specific goals such as restaurant booking, hotel reservation, and flight search. many popular datasets have been introduced to facilitate the dialog research, ranging from simple single-domain @xcite to more difficult multi-domain dialogs @xcite . similar to the widely studied glue @xcite and superglue @xcite for general natural language understanding, largescale benchmarks have been specifically designed for dialog systems. for instance, @xcite proposed the dialoglue, a collection of seven popular english dialog datasets, to assess the natural language understanding ability in a few-shot setting. more recently, @xcite introduced the raddle, another large-scale benchmark that includes a diagnostic checklist facilitating detailed robustness analysis for pre-trained language models @xcite . however, compared to the abundance of english dialog benchmarks, the resource of chinese dialog datasets in goal-oriented dialog remains quite limited, let alone the large-scale benchmark suitable for the dialog evaluation of chinese pre-trained language models. moreover, current chinese dialog datasets, such as crosswoz @xcite and risawoz @xcite , mainly focus on the schema-guided dialog as in the classical multi-woz dataset @xcite , where conversations are grounded on a table-formed ontology. this kind of ontology usually defines a set of possible slot-value pairs to be recognized during the dialog for database query @xcite . in real-world applications, however, more types of goal-oriented dialogs are involved. for instance, in @xcite the conversational process is grounded on a pre-defined tree-formed ontology, where the whole process is constrained in a dialog flow. also, in some scenarios, the conversation has been simplified into a multi-turn response selection problem so that the dialog system is able to transfer to new domains quickly and reliably @xcite @xcite . these observations motivate us to construct cgodial, a new large-scale chinese benchmark for goal-oriented dialog, targeting the evaluation of model adaptability and robustness in real situations. concretely, we consider three types of goaloriented dialogs with different knowledge database sources. the first is the slot-based dialog (sbd), where a system often asks a user for required attributes to constrain the search in a table-formed database and then provides entities and extra information to the user. to build sbd, we adapt the existing chinese dataset risawoz @xcite by supplementing it using crowd-sourcing for more difficult variations such as noisy expression augmentation, external knowledge utilization, and out-of-scope management. 44,800 new dialogs spanning 12 domains are collected for sbd. the second is the flow-based dialog (fbd), where the system guides the user to fulfill specific tasks based on tree-structured dialog flow. this flow is a new form of knowledge database that defines causal constraints of conversations. for example, when a user comes to withdraw money in a housing insurance consulting scenario, the system must first determine the user's identity before moving to the next step. therefore, compared with sbd, fbd has a strict order to request user information we collect new dialogs between human customers and an online customer service dialog agent from real businesses, which to the best of our knowledge, is the first chinese fbd dataset. after data desensitization, we have 6,786 dialogs and 45,601 turns spanning four different domains in the end. the third is the retrieval-based dialog (rbd), like babi-dialog @xcite , where the dialog system learns to select the correct response from a candidate response set. we build rbd by adapting the existing ecd @xcite dataset and adding more noisy user expressions like asr errors to raise the overall difficulty. we choose 56,000 non-ambiguous complex dialog examples from the original dataset. to fully assess dialog model capabilities in general prediction, fast adaptability, and reliable robustness. for all the above types of dialog datasets, we propose four different experimental settings, including the combinations of training with either the full training set or a few-shot training set, and testing with either the standard test set or a hard test subset. extensive experiments have been conducted on cgodial with various chinese pre-trained language model baselines for dialog modeling, such as chinese-t5 @xcite and cdial-gpt @xcite . to further facilitate the research of dialog pre-training, we also release a new unilm-based dialog model pretrained on large-scale human-to-human dialog corpora, which achieves the best results on all tasks. therefore, the contributions of this paper are three-fold: ## cgodial benchmark in this section, we elaborate on the three datasets in our cgodial benchmark, in terms of task definition and dataset construction. to give an overall impression, figure 1 illustrate typical examples for each dataset and ## conclusion we propose cgodial, a new large-scale chinese goal-oriented dialog benchmark featuring three different types of datasets: slot-based dialog, flowbased dialog, and retrieval-based dialog, which are used for comprehensive dialog evaluation. we also propose three different experimental settings: standard training, limited data use, and robustness testing, to assess the dialog model from the aspects of general prediction, fast adaptability, and reliable robustness, respectively. results from several competitive baselines show the challenge of cgodial, which is worthy of follow-up research.