Semi-Supervised Knowledge-Grounded Pre-training for Task-Oriented Dialog Systems (2022)

Abstract: Recent advances in neural approaches greatly improve task-oriented dialogue (TOD) systems which assist users to accomplish their goals. However, such systems rely on costly manually labeled dialogs which are not available in practical scenarios. In this paper, we present our models for Track 2 of the SereTOD 2022 challenge, which is the first challenge of building semisupervised and reinforced TOD systems on a large-scale real-world Chinese TOD dataset MobileCS. We build a knowledge-grounded dialog model to formulate dialog history and local KB as input and predict the system response. And we perform semi-supervised pretraining both on the labeled and unlabeled data. Our system achieves the first place both in the automatic evaluation and human interaction, especially with higher BLEU (+7.64) and Success (+13.6%) than the second place.

## introduction task-oriented dialogue (tod) systems assist users to accomplish their goals like booking a ticket and make an effect on everyone's lives with recent advances in neural approaches @xcite . a typical tod system consists of three sub-modules: to facilitate relevant research, seretod 2022 workshop foot_1 proposes the first challenge of building semi-supervised and reinforced tod systems by releasing a large-scale chinese tod dataset mo-bilecs from real-world dialog transcripts between real users and customer-service staffs from china mobile. mobilecs contains 10,000 labeled dialogs and 90,000 unlabeled dialogs. there are two tracks: (1) information extraction (track 1) aims to extract entities together with their slot values. (2) taskoriented dialog system (track 2) aims to build a complete tod system, including predicting the user intent, querying the local kb, and generating appropriate system intent and response according to the given dialog history. the core challenge is how to combine a small labeled dataset and a large unlabeled dataset. in this paper, we present our system for track 2 of the seretod 2022 challenge. the main intuition behind our system comes from semi-supervised knowledge-grounded pre-training on both labeled and unlabeled datasets. we divide track 2 into two task groups, classification (user intent and system intent) and generation (system response). for the classification tasks, we employ roberta-large 3 and build two separate classification models. we also perform continual pre-training on all the dialog data. for the generation task, we build a knowledge-grounded dialog model, which is the key point of this paper. specifically, we firstly use a1: "飞享十八的是吧" u1:"嗯你好你给我办理一个十八块钱的套餐可 以吗" ## task description mobilecs is a large chinese tod dataset collected from real-world dialog transcripts between real users and customer-service staffs. different from the simulated multiwoz dataset @xcite , it consists of real-life data and large unlabeled dialogs. specifically, mobilecs contains 10,000 labeled dialogs and 90,000 unlabeled dialogs. the full data statistics are shown in to measure the performance of tod systems, both automatic evaluation and human evaluation will be conducted. for automatic evaluation, metrics include precision/recall/f1 score, success rate and bleu @xcite score. p/r/f1 are calculated for both predicted user intents and system intents. success rate is the percentage of generated dialogs that achieve user goals. bleu score evaluates the fluency of generated responses 6 . for human evaluation for different tod systems, real users will interact with those systems according to randomly given goals. for each dialog, the user will score the system on a 5-point scale (1-5) by the following 3 metrics. 5 denotes the best and 1 denotes the worst, respectively. ## conclusion in this paper, we present our models for track 2 of the seretod 2022 challenge aiming to build semisupervised and reinforced tod systems. we divide track 2 into two task groups, classification (user intent and system intent) and generation (system response). for the classification tasks, we employ roberta-large and build two separate classification models. we also perform continual pre-training, class-wise threshold, and adversarial training strate-