MVP : Multi-task Supervised Pre-training for Natural Language Generation (2023)

Abstract: Pre-trained language models (PLMs) have achieved remarkable success in natural language generation (NLG) tasks. Up to now, most NLG-oriented PLMs are pre-trained in an unsupervised manner using the large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with labeled data (i.e. “supervised pre-training”) showcase superior performance compared to unsupervised pre-trained models. Motivated by the success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation. We collect a large-scale natural language generation corpus, MVPCorpus, from 77 datasets over 11 diverse NLG tasks. Then we unify these examples into a general text-to-text format to pre-train the text generation model MVP in a supervised manner. For each task, we further pre-train specific soft prompts to stimulate the model’s capacity to perform a specific task. Our MVP model can be seen as a practice that utilizes recent instruction tuning on relatively small PLMs. Extensive experiments have demonstrated the effectiveness and generality of our MVP model in a number of NLG tasks, which achieves state-of-the-art performance on 13 out of 17 datasets, outperforming BART by 9.3% and Flan-T5 by 5.8%.

## introduction natural language generation (nlg, also known as text generation) is a crucial capacity for language intelligence, which aims to generate human-like texts on demand @xcite . since the emergence of the pre-training and fine-tuning paradigm, pre-trained language models (plms) have dominated mainstream approaches for nlg tasks @xcite . with a large-scale general corpus, the majority of plms are pre-trained in an unsupervised (self-supervised) manner by leveraging intrinsic data correlations as corresponding author supervision signals. however, unsupervised pretraining is likely to incorporate noise that affects the performance of downstream tasks @xcite , also leading to a slower rate of acquiring knowledge @xcite . in the meanwhile, more and more large-scale labeled datasets have become easily accessible @xcite . there is growing evidence that pre-training with labeled data can further improve the performance of plms, both in the fields of computer vision @xcite and natural language processing @xcite . these promising developments motivate us to consider pre-training text generation models with labeled data, which is called "supervised pretraining" @xcite . existing work has shown that supervised pre-training can explicitly learn task-specific characteristics and alleviate the discrepancy between unsupervised pre-training and supervised fine-tuning @xcite . furthermore, most nlg systems are often trained in a supervised way, requiring supervision signals to learn the input-to-output transformation. for example, dialogue systems learn to generate appropriate responses based on historical utterances, and text summarization systems learn to extract essential information from long documents according to human-written summaries. therefore, we suspect that supervised pre-training is more suited for nlg-oriented plms in essence since it can provide task-related instructions early in the pre-training stage instead of a later fine-tuning stage. inspired by the recent success of supervised pre-training, we propose multi-task supervised pre-training (mvp) for natural language generation by leveraging a variety of labeled text generation datasets. specially, we collect a largescale labeled corpus, mvpcorpus, consisting of 77 datasets over 11 text generation tasks. since recent research shows that an extensive scale of settings supervised pre-training unsupervised pre-training nlg mvp (ours) gpt-2, mass, bart, t5 nlu flan, t0, muppet, ext5 bert, xlnet, roberta, t5 to develop our approach, we adopt a transformer-based @xcite sequenceto-sequence model as the backbone. in multi-task training, different tasks may "neutralize" the ability learned through other tasks @xcite . to mitigate this potential issue, we propose to learn task-specific prompts based on the mvp model, following the structure of prefix-tuning @xcite . task-specific pre-training enables prompts to "store" specialized knowledge for each corresponding task. integrating mvp with task-specific prompts can further stimulate the model's capacity to perform some specific tasks. to summarize, our main contributions center around the following research questions: our zero-shot performance also surpasses t0-11b @xcite by a large margin. furthermore, the experiments on unseen nlg and nlu tasks demonstrate that our supervised mvp model has a strong generality for unseen tasks. for reproducing and reusing our work, we release the mvpcorpus collection, all the mvp model variants, and accordingly codes at the link: https://github.com/rucaibox/mvp . multi-task learning. our pre-training process is also related to multi-task learning (mtl), a method of mixing multiple tasks into a single training process @xcite . a model trained with mtl can benefit from helpful knowledge of relevant tasks, resulting in improved performance @xcite . recently, mt-dnn @xcite and muppet @xcite collect tens of datasets in the multi-task procedure and achieve better performance in downstream tasks. the pre-finetuning schema proposed in muppet shares a similar idea with our study. @xcite further combine the denoising pre-training task of t5 @xcite and multi-task learning to pre-train a new model, ext5. mtl has also contributed to sub-fields of text generation, such as open-ended dialogue system @xcite , task-oriented dialogue system @xcite ), text style transfer @xcite , and question answering @xcite . at the same time, researchers explore the transferability of models trained on multi-task datasets @xcite . flan @xcite , t0 @xcite , zeroprompt @xcite , and flan-t5 @xcite investigate the zero-shot or few-shot generalization abilities of large language models (llms) @xcite trained on numerous task datasets with well-designed prompts. compared with these works, we aim to explore multi-task learning to derive both effective and general nlg models in a supervised pre-training manner. prompt learning. prompt learning is a thriving method in the field of nlp. prompt learning converts fine-tuning text into a format similar to pre-training to leverage implicit pre-training knowledge and alleviate the discrepancy between pretraining and fine-tuning @xcite . gpt-2 @xcite and t5 @xcite add human-written task prompts to the input text. for instance, t5 prepends "summarize:" to the input document for summarization tasks. some researchers also design elaborate prompts for each task and dataset and investigate their effectiveness and robustness @xcite . to overcome the constraints of manually constructed prompts, researchers develop continuous (soft) prompts that can be optimized in continuous space @xcite @xcite . considering the random initialization of soft prompts, @xcite propose ppt to pre-train continuous prompts using unlabeled data. spot @xcite , unified-skg @xcite , and ptg @xcite further learn the prompts on related tasks and transfer the prompts to new tasks. ## the mvp model this section introduces our mvp model: a multitask supervised pre-trained model for natural language generation. the overview of our model is illustrated in figure 1 . ## experiment results in this section, we mainly investigate the effectiveness and generality of our mvp model. we conduct extensive experiments in different settings: for the full tuning setting (tables 2 and 11 ), we fine-tune the entire model (including the backbone mvp and prompts), while for the parameterefficient tuning ( ## discussion differences with existing methods. to the best of our knowledge, existing supervised pre-training works mainly focus on nlu tasks @xcite or a small number of nlg tasks @xcite . given the superior performance achieved by supervised pre-training approaches, it is important to explore supervised pre-training for deriving both effective and general nlg models. our work makes a significant contribution in this direction, achieving sota performance with a single model on 13 of 17 datasets. compared with its strong counterpart, ext5 @xcite , our mvp model outperforms it in 26 out of 27 metrics (detailed in appendix c.2). in order to better understand the difference between our work and previous supervised (multi-task) pre-training studies, we present a detailed comparison in applicability. to facilitate the application of our work, we have released the collection corpus, pretrained models, task-specific prompts, and generated texts. our collected mvpcorpus is the largest nlg task collection, which can be a high-quality resource for recent llms @xcite . we can use all the data to pre-train a general model or select a subset to continue pre-training a domain-or task-specific model @xcite our mvpcorpus can also be considered as the evaluation benchmark for different nlg tasks. furthermore, our mvp model can be employed to achieve competitive results in various nlg tasks. users can fine-tune the mvp model or integrate it with task-specific prompts based on sufficient labeled data. notably, our mvp model can be directly employed to obtain good performance in zero-shot learning. in addition, our mvp model can provide effective parameter initialization for improving existing methods, as described in section 4.3. finally, the task-specific prompts and the generated texts can be further used to study the task similarity and their effect on the multi-task pre-training. ## conclusion in this paper, we present multi-task supervised pre-training (mvp) for natural language generation. firstly, we collect a large-scale nlg corpus, mvpcorpus, from 77 datasets over 11 diverse nlg tasks. after converting various nlg tasks into a unified text-to-text format, we propose multi-task supervised pre-training to learn an effective and general model mvp with task-specific prompts for nlg tasks. extensive experiments have demonstrated that: (1) supervised pre-training is beneficial for nlg tasks as an effective solution. our mvp model outperforms its strong counterparts bart and flan-t5 and even achieves sota performance on 13 out of 17 datasets; (2) supervised pre-trained models have strong generality on unseen generation or even understanding tasks. in future work, we will explore the multilingual version of our mvp model by covering more datasets in other languages. such a model is expected to capture language-independent task characteristics and improve generation tasks in the minority language. besides, it is interesting to study how different tasks relate to each other in the unified semantic space, which can inspire methods that incorporate task relations as prior.