DSPM - NLG : A Dual Supervised Pre-trained Model for Few-shot Natural Language Generation in Task-oriented Dialogue System (2023)

Abstract: In few-shot settings, fully conveying the semantic information of the dialogue act is a crucial challenge for Natural Language Generation (NLG) in the task-oriented dialogue system. An interesting fact is that NLG and Spoken Language Understanding (SLU) are a natural dual problem pair. Suppose the response generated by the NLG module can be restored to the corresponding dialogue act by the SLU module, which reflects that the generated response fully conveys the semantic information of the dialogue act. Based on this idea, a novel Dual Supervised Pre-trained Model for a few-shot Natural Language Generation (DSPM-NLG) is proposed to regularize the pre-training process. We adopt a joint model with a dual supervised framework to learn the dual correlation between NLG and SLU from the perspective of probability. In addition, a slot-masked strategy is designed to enable the model to focus better on the key slot-value pairs. DSPM-NLG is continuously trained on existing public large-scale annotated data, which thoroughly learns the duality between two tasks to enhance the semantically controlling and generalization abilities of the pre-trained model. Experiments demonstrate that our proposed model performs outstandingly on the few-shot benchmark dataset and outperforms the previous SOTA results.

## introduction task-oriented dialogue systems have been demonstrated to be effective in aiding users accomplish various tasks in multiple domains, such as airline ticket booking, restaurant and hotel reservations. ## related work existing nlg models can be mainly summarized into two major categories. (1) template-based nlg models @xcite generate responses according to manually developed rules. these models generate responses that can convey the semantics information of certain predefined dialogue acts. nevertheless, the handcraft templates are difficult to cover potentially unforeseen dialogue acts, and the generated response is not always natural. (2) statistical-based nlg models @xcite @xcite @xcite @xcite generate responses via training from massive annotated data. with the rise of attention mechanism, more approaches have been proposed, e.g., hierarchical attention network @xcite @xcite . and then, some nlg works adapted a multi-task learning framework to improve the performance @xcite . in particular, some scholars exploit the relationship between slu and nlg to improve the performance of two tasks @xcite @xcite @xcite . subsequently, many works introduce pre-trained models @xcite @xcite @xcite @xcite such as gpt2, and the overall performance of nlg is greatly improved. recently, to deal with the challenge of few-shot learning, data augmentation has been widely applied to nlg. @xcite proposed sc-gpt model. they pre-train gpt with large-scale nlg corpus collected from publicly available dialogue datasets and then fine-tuned the model on the target domain with few training instances. @xcite proposed a data augmentation approach that constructed dialogue acts and responses from the open-domain dialogues and applied the new data to sc-gpt. compared with previous work, we try to explore the duality between slu and nlg in the pre-training stage. the difference between the proposed model and the previous methods is mainly reflected in the following two aspects: first, dual supervised learning is only applied in the pre-training. thus, in few-shot settings, our model does not require any slu annotated data and does not increase additional computation in fine-tuning and inference stages. it is worth mentioning that our model also avoids the error transfer between slu and nlg in the inference stage. second, in the pre-training stage, we collect a large amount of labeled data for slu and nlg. the training of a large amount of labeled data enables the pre-trained model to have a strong semantically controlling ability rather than just learning the relationship between the two tasks in some specific domains to improve the performance of both tasks. obtain the maximum likelihood estimation of x i from the dual input y i via the dual task: the probabilistic duality constraint is incorporated: ## experimental setup dataset comparative experiments are conducted on the publicly available datasets for nlg, namely, fewshotwoz @xcite and few-shotsgd @xcite , respectively. the two datasets include seven domains and sixteen domains, respectively 2 . compared with the other existing datasets, they have several favorable properties for few-shot learning: more domain, fewer training instances, and lower training overlap. for the fewshotwoz, each domain has 50 training instances, and the average number of test instances is 472.857. the overlap percentage is 8.82%. since slu has been introduced in the model, labels required for the slu tasks are added to the standard nlg dataset in the pre-training stage. we obtain labeled data of the slu according to the dialogue acts by the matching method. ## results and analysis we compare our model with previous state-of-theart models. the overall results of nlg experiments on the fewshotwoz dataset are shown in fewshotsgd are shown in ## conclusion in this paper, we proposed a novel dual supervised pre-trained model for nlg. we explore the duality between slu and nlg from the perspective of joint probability in the pre-training stage. the slot-masked strategy is designed to constrain the dspm-nlg model to focus on the slot-value pairs in dialogue acts.