Z ero P rompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization (2022)

Abstract: We propose a multitask pretraining approach ZeroPrompt for zero-shot generalization, focusing on task scaling and zero-shot prompting.While previous models are trained on only a few dozen tasks, we scale to 1,000 tasks for the first time using real-world data. This leads to a crucial discovery that task scaling can be an efficient alternative to model scaling; i.e., the model size has less impact on performance with an extremely large number of tasks. Our results show that task scaling can improve training efficiency by 30 times in FLOPs.Empirically, ZeroPrompt substantially improves both the efficiency and the performance of zero-shot learning across a variety of academic and production datasets.

## introduction recent progress like gpt-3 @xcite demonstrates the possibility of prompting on largerscale models for zero-shot learning, but the performance of zero-shot generalization still falls short on many tasks compared to fully-supervised finetuning. further, other works proposed to include a set of supervised tasks into pretraining @xcite @xcite , and prompts are often used in the framework to unify the tasks. @xcite converted different datasets into a unified "yes/no" question answering format with label descriptions. @xcite extended the scope by considering more task types and a larger model. t0 @xcite collected a large set of diverse prompts for each task to further enhance performance. despite the effects of model scaling and prompts scaling @xcite have been explored, only dozens of training tasks are exploited in these works. it is still not clear how scaling the number of training tasks to hundreds even thousands of tasks affects the performance of multitask pretraining. we hypothesize that task scaling plays an important role in training generalizable zero-shot systems and explore the limits of task scaling using 1,000 tasks. interestingly, our empirical study reveals that task scaling can be an efficient alternative to model scaling, as shown in figure 1 . with an extremely large number of training tasks, the model size has less impact on performance. a 0.4b model can achieve comparable zero-shot performance to that of a 12b model, improving training efficiency by 30 times in terms of flops and the serving efficiency as well. our contributions can be summarized as follows. ## related work pretrained language models, like bert @xcite , xlnet @xcite , t5 @xcite and gpts @xcite , have achieved strong performance on various nlp tasks. in some cases, pretrained models can perform well with only a few training samples @xcite , or even without any training sample @xcite . it has been shown that augmenting unsupervised pretraining with supervised data can significantly improve task performance during finetuning @xcite . some recent studies followed this idea and obtained improved few-shot or zero-shot generalization in the same manner. for instance, mishra et al. @xcite built a dataset with task instructions, and crossfit @xcite introduced a repository of few-shot text-to-text tasks. @xcite and t0 @xcite applied instruction-tuning of many tasks with 137b and 11b parameters, respectively. ext5 @xcite applies multitask pretraining as well, but it focuses on multitask cotraining transfer instead of zero-shot generalization. our zeroprompt utilizes labeled data in the pretraining phase, and we aim at studying the task scaling law of zero-shot generalization by adopting 1,000 real-world tasks. ## zeroprompt we follow the same framework of multitask zeroshot learning in @xcite , where models are pretrained on a variety of tasks and then tested on held-out unseen tasks. ## conclusions in this paper, we propose zeroprompt, a multitask prompted pretraining method that significantly improves the zero-shot generalization ability of language models. in our experiments, we collect over 1,000 real-world production tasks to study the task scaling law. we find that on our considered datasets, the zero-shot performance gap between small and large models becomes less significant when having more training tasks. as a result, task scaling can substantially improve training and serving efficiency.