Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers (2023)

Abstract: This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5. We study various designs to pretrain T5 using an auxiliary model to construct more challenging token replacements for the main model to denoise. Key aspects under study include the decoding target, the location of the RTD head, and the masking pattern. Based on these studies, we develop a new model, METRO-T0, which is pretrained using the redesigned ELECTRA-Style pretraining strategies and then prompt-finetuned on a mixture of NLP tasks. METRO-T0 outperforms all similar-sized baselines on prompted NLP benchmarks, such as _T0 Eval_ and MMLU, and rivals the state-of-the-art T0-11B model with only **8%** of its parameters. Our analysis on model’s neural activation and parameter sensitivity reveals that the effectiveness of METRO-T0 stems from more balanced contribution of parameters and better utilization of their capacity. The code and model checkpoints are available at [ https://github.com/gonglinyuan/metro_t0 ]( https://github.com/gonglinyuan/metro_t0 ).

## introduction recent work in nlp has shown that pretrained language models have made noteworthy progress toward generalization to unseen tasks. despite being pretrained on only language modeling objectives, large language models can perform reasonable zeroshot generalization given natural language instructions, i.e. prompts @xcite . further research shows that finetuning language models on a mixture of tasks with prompt templates enhances their performance on held-out new tasks @xcite . in recent years, two significant research paths have emerged in the field of pretrained language models: one seeks to improve generalization either by scaling up the model, increasing parameters, data, and compute, or by refining prompts. another divergent yet complementary approach focuses on augmenting the efficiency of pretraining, particularly in the context of bert-style models. this approach has been proven to significantly improve pretraining efficiency through the use of model-generated pretraining signals, as evidenced by electra @xcite , coco-lm @xcite , and metro-lm @xcite . however, this improvement has primarily been witnessed in single-task supervised finetuning settings. our work seeks to bridge these two areas of research. we present a novel method that enhances the pretraining efficiency of t5, a widely used encoder-decoder transformer in prompt-based learning, by utilizing electra-style model-generated signals. our preliminary studies, however, encountered many challenges in pretraining t5 with modelgenerated signals, particularly in designing an effective objective to train the decoder and ensuring training stability. to address these challenges, we study the impact of key components in this pretraining scheme, such as the decoding target, the location of the replace token detection (rtd) task, and the masking pattern. then we redesign the pretraining algorithm to solve training stability issues, thus bringing in the benefits of electrastyle pretraining to t5-style transformer encoderdecoder models. the pretrained model is then finetuned on a family of multi-task training mixtures of nl-prompted dataset, which has previously been used to train the t0 models @xcite . our model, metro-t0, is a t0 model pretrained with model generated denoising training objective. experimental results show that metro-t0 is highly parameter efficient. it consistently outperforms similar-sized baselines on all nl-prompted benchmark we evaluated upon. as shown in figure 1, metro-t0 base++ outperforms t0 3b @xcite with only 7% of its parameters on the t0 eval benchmark. moreover, metro-t0++ large++ rivals 14x larger t0++ 11b , the stateof-the-art in prompt-based learning. our method is also compute efficient: metro-t0 pretrained for 500k steps has similar performance as its t0 counterpart pretrained for 2m steps. to further understand the benefit of metro pretraining, we conduct two studies on the pretrained metro-t0 model, analyzing its neural activation and parameter sensitivity. the studies show that model-generated signals balance the contribution of each nn parameter and reduce the number of under-activated neurons by 55%, indicating that a key source of the improved pretraining efficiency is better utilization of network parameters. ## related work prompt-based learning with language models. prompt-based learning allow language models to handle a wide range of tasks with no training data (zero-shot) or a few training data (few-shot), by leveraging natural language instructions and task demonstrations as context @xcite . @xcite proves the effectiveness of prompt-based learning as a framework of multi-task learning for text-to-text transformers such as t5. lms are usually finetuned with nl instructions to improve their performance and usability. such a procedure is called promptfinetuning. the finetuning data comes from aggregated mixtures of nlp tasks @xcite , dialogs @xcite , or even chain-of-thoughts @xcite . our work aims to improve the zero-shot generalization of t5like text-to-text lms in prompt-based learning by efficient and effective pretraining strategies. efficient pretraining using model-generated signals. training big language models require sub-stantial computational resources. this paper is part of a line of research that improves the pretraining efficiency of lms using model-generated signals, i.e., metro @xcite , pioneered by electra @xcite , a transformer encoder pretrained using signals generated by an auxiliary bert. various studies @xcite @xcite show that an auxiliary model can generate informative training signals that greatly improve the efficiency and effectiveness of bert-like transformer encoder models, as evaluated on supervised single-task benchmarks like glue @xcite . compared with these works, we use modelgenerated signals to pretrain t5-like transformer encoder-decoder models and evaluate this model on large-scale nl-prompted benchmarks. ## preliminaries this section provides an overview of t5 and metro-style pretraining. ## experimental setup model architecture. each of our models has an architecture similar to t5 @xcite . we train models in three standard setups: base, base++, and large++. our base/base++ model has an architecture similar to t5 base . our large++ model has an architecture similar to t5 large except for some differences mentioned in section 4. the auxiliary model for generating training signals is a transformer encoder of the same hidden size as the main model but is shallower: it consists of 4 layers in base/base++ and 6 layers in large++. we follow @xcite and share token embeddings between the main and the auxiliary model. pretraining. our base model is pretrained on english wikipedia and bookcorpus (16gb of texts) for 131 billion tokens (512 tokens per sequence, 2,048 sequences per batch, and 125k steps). base++/large++ is the training configuration first used in roberta @xcite : pretraining on a mixed corpus of 160gb texts for a maximum 2.1 trillion tokens (512 tokens per sequence, 2,048 sequences per batch, and at most 2m steps). prompt-finetuning. we finetune each of our pretrained metro-t5 models on three multi-task mixtures: t0/t0+/t0++ train, using the same prompt templates and shuffling strategy as @xcite does. each model is finetuned for 125k steps, using the same hyperparameters as pretraining, except the peak learning rate is reduced to 0.1x. we do not perform any checkpoint selection and simply use the last checkpoint at 125k steps for evaluation. evaluation. we evaluate zero-shot generalization on the t0 eval benchmark @xcite and the massive multi-task language understanding (mmlu) benchmark @xcite . t0 eval consists of 11 datasets in natural language inference, coreference, word sense disambiguation, and sentence completion. mmlu includes exam questions from 57 tasks such as maths, history, law, and medicine. for each dataset, we report accuracy on the validation split. following gpt-3 @xcite and t0 @xcite , we use rank classification for inference. for t0 eval, we use the same prompt templates as t0. for mmlu, we use prompt templates from the ai2 reasoning challenge (ai2-arc) @xcite , concatenated with 5 passages retrieved using t5-ance @xcite (see appendix a.8 for details). when there are multiple prompts for a dataset, we do not perform prompt selection based on the validation split, because such prompt selection will break the "zeroshot" evaluation. instead, we report the average accuracy across all prompts for this dataset, following the standard practices of @xcite . baselines. for a fair comparison, the main baseline is our own t0 runs. except for metro-style pretraining, our t0 baselines use the same transformer architecture, pretraining data, and promptfinetuning data, pretrained in the same computational environment. we also compare with the reported numbers of other language models that supports zero-shot prompting, including pretraining-only models such as gpt-3 @xcite and t5 @xcite , as well as prompt-finetuned models such as t0 @xcite and flan-t5 @xcite . t0/t0+/t0++ is pretrained on the the c4 @xcite corpus of 800gb of texts for 1 trillion tokens and then prompt-finetuned on the t0/t0+/t0++ train multitask mixture af- all reported results use accuracy averaged over 57 subtasks as their metric. ## conclusion this paper presents a new method for improving the zero-shot generalization of t5-like text-to-text transformers by incorporating model-generated signals in the pretraining process. metro-t0, the model sufficiently trained using our redesigned pretraining method, is highly parameter efficient and compute efficient. we hope that the success of our approach could inspire further work on efficient big lm pretraining and prompt-based learning.