P rophet N et- X : Large-Scale Pre-training Models for E nglish, C hinese, Multi-lingual, Dialog, and Code Generation (2021)

Abstract: Now, the pre-training technique is ubiquitous in natural language processing field. ProphetNet is a pre-training based natural language generation method which shows powerful performance on English text summarization and question generation tasks. In this paper, we extend ProphetNet into other domains and languages, and present the ProphetNet family pre-training models, named ProphetNet-X, where X can be English, Chinese, Multi-lingual, and so on. We pre-train a cross-lingual generation model ProphetNet-Multi, a Chinese generation model ProphetNet-Zh, two open-domain dialog generation models ProphetNet-Dialog-En and ProphetNet-Dialog-Zh. And also, we provide a PLG (Programming Language Generation) model ProphetNet-Code to show the generation performance besides NLG (Natural Language Generation) tasks. In our experiments, ProphetNet-X models achieve new state-of-the-art performance on 10 benchmarks. All the models of ProphetNet-X share the same model structure, which allows users to easily switch between different models. We make the code and models publicly available, and we will keep updating more pre-training models and finetuning scripts.

## introduction in recent years, quite a few natural language generation pre-training models are proposed @xcite @xcite . downstream generation tasks benefit from these large scale pre-training models greatly in fluency and accuracy. researchers also extend these general pre-training works into specific domains such as dialogpt @xcite is extended from @xcite for dialog system, mbart @xcite is extended from bart @xcite for multi-lingual generation, codebert @xcite is extended from bert @xcite for programming language modeling, etc. although there are pre-trained models for some specific domains, it is not convenient for users to find them and set them up. besides, even some models in the same pre-training family with the same model structure and pre-training tasks, their codes and details vary a lot because of different implementation and backends selection. prophetnet @xcite is firstly proposed as an english text pre-training model with future tokens' prediction, and successfully improves the performance on different downstream nlg tasks. in this work, we pre-train the prophetnet on different corpus, respectively. the corpus covers different languages and domains. all the pre-trained models share the same model structure with different vocabularies. we provide six pre-trained models with downstream task finetuning scripts, including prophetnet-en pre-trained with 160gb english raw text, prophetnet-zh pre-trained with 160gb chinese raw text, prophetnet-multi with 101gb wiki-100 corpus and 1.5tb common crawl foot_0 data, prophetnet-dialog-en with 60 million sessions reddit open-domain dialog corpus, prophetnet-dialog-zh with collected chinese dialog corpus over 30 million sessions, and prophetnet-code pre-trained with 10 million codes and documents. prophetnet-x achieves new state-of-the-art results on 10 benchmarks, including chinese summarization (matinf-summ @xcite and lc-sts @xcite ), chinese question answering (matinf-qa @xcite ), cross-lingual generation (xglue ntg @xcite and xglue qg @xcite ), english summarization (msnews @xcite ), english dialog generation (dailydialog @xcite , personachat @xcite , and dstc7-avsd @xcite ), and code summarization (codexglue @xcite ). users can simply download the prophetnet-x repository and find corresponding pre-trained model with downstream task finetuning scripts. the main contributions of prophetnet-x can be described as follows: ## related work prophetnet @xcite is the most related to our work since we carry out pre-training based on it. other related works involve pre-training works in different domains. for english gener-ation pre-training, mass @xcite proposes an unsupervised pre-training task with span masked and recover. bart @xcite feeds corrupted sentences into the encoder and reconstructs the original sentences. gpt @xcite models perform language modeling pre-training with transformer decoder. for multilingual pre-training, mbart @xcite introduces language labels to adopt bart denoising pre-training. based on gpt @xcite , dialogpt @xcite and cdial-gpt @xcite adopts language model pre-training with english and chinese dialog corpus respectively. codebert @xcite and @xcite are two pre-training models for programming languages. @xcite is similar to multilingual bart with language tags to perform denoising pre-training on programming languages. ## conclusion in this paper, we pre-train prophetnet-x on various languages and domains, including open-domain (for english, chinese, and multi-lingual), dialog (for english and chinese), and programming (for ruby, javascript, go, python, java, and php). all the models share the same model structure and are easy to use. extensive experiments show that prophetnet-x achieves new state-of-the-art performance on 10 benchmarks. in the future, we will extend prophetnet-x to support more domains such as biomedical text and protein pre-training.