PLATO - XL : Exploring the Large-scale Pre-training of Dialogue Generation (2022)

Abstract: To explore the limit of dialogue generation pre-training, we present the models of PLATO-XL with up to 11 billion parameters, trained on both Chinese and English social media conversations. To train such large models, we adopt the architecture of unified transformer with high computation and parameter efficiency. In addition, we carry out multi-party aware pre-training to better distinguish the characteristic information in social media conversations. With such designs, PLATO-XL successfully achieves superior performances as compared to other approaches in both Chinese and English chitchat. We further explore the capacity of PLATO-XL on other conversational tasks, such as knowledge grounded dialogue and task-oriented conversation. The experimental results indicate that PLATO-XL obtains state-of-the-art results across multiple conversational tasks, verifying its potential as a foundation model of conversational AI.

## introduction the efficacy of the pre-training paradigm, where large-scale transformer models are trained with massive plain texts, has been widely recognized in natural language processing @xcite . to further boost the performance of these language models, there is a trend to enlarge the model size, dataset size, and the amount of compute used for training @xcite . particularly, the gpt-3 model with 175b parameters demonstrates strong zeroshot or few-shot learning capacities without taskspecific fine-tuning on downstream tasks @xcite . distinct from the general language models, dialogue generation models are usually pre-trained with human-like conversations collected from social media. dialogpt @xcite at- * equal contribution. in this paper, we argue that the conversation quality may keep benefiting from the enlarged model scale with appropriate pre-training designs. to this end, we explore the large-scale pre-training of dialogue generation models with up to 11b model parameters, namely plato-xl. to train such a large model, we adopt the architecture of unified transformer with high computation and parameter efficiency. in addition, we carry out multi-party aware pre-training to better distinguish the characteristic information in social media conversations. with such designs, plato-xl achieves superior performances as compared to other approaches in both chinese and english chitchat. more specifically, plato-xl shows a strong capability of absorbing common knowledge within its huge parameters; therefore, it is able to alleviate the wellknown hallucination problem 1 . besides, thanks to the multi-party aware pre-training, plato-xl effectively reduces the inconsistency phenomenon in multi-turn conversations. in addition to open-domain chitchat discussed above, there are two other common conversational tasks @xcite : knowledge grounded dialogue, and task-oriented conversation. in the experiments, we also explore the ability of plato-xl as the foundation model of conversational ai. our experimental results indicate that plato-xl is able to outperform other dialogue generation models across multiple conversational tasks. we have released our source code together with the english model at github foot_2 , hoping to facilitate frontier research in dialogue generation. ## conclusion in this paper, we explore the large-scale pretraining of dialogue generation and present the 11 billion parameter model of plato-xl. experimental results demonstrate that plato-xl achieves superior performance as compared with other approaches in both chinese and english chitchat. particularly, the problems of hallucination and inconsistency are alleviated remarkably in plato-xl, mainly attributed to the implicit knowledge absorbed in the tremendous parameters and the multi-party aware pre-training. besides the open-domain conversation, plato-xl obtains state-of-the-art results on multiple knowledge grounded and task-oriented conversations, verifying its capacity as a foundation model of conversational ai.