M uppet: Massive Multi-task Representations with Pre-Finetuning (2021)

Abstract: We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g. RoBERTa) and generation models (e.g. BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.

## introduction the recent success of language model pre-training @xcite @xcite @xcite is remarkable, at least in part, due to the exclusive use of self supervision, without any manually labeled data. for many tasks, however, we already have training examples for related problems, which we should be able to leverage. recent work has shown gains from fine-tuning schemes that are multi-task @xcite and multi-stage @xcite , but it can be difficult to know which intermediate tasks will best transfer @xcite . in this paper, we show that multi-task supervised tuning, if done at a sufficiently large scale with many different tasks, can be an effective second stage of task-agnostic pre-training, removing the need to pre-select the best intermediate tasks. more specifically, in addition to the standard pre-training/fine-tuning methodology of learning language tasks, we introduce a new intermediate stage, pre-finetuning. pre-finetuning involves a massive multi-task learning step (4.8 million total training examples) performed on around 50 classification, summarization, question answering, and common sense reasoning tasks. we believe we are the first to investigate multi-task learning at this scale in terms of both number and types of tasks. we show, in particular, that standard multi-tasking schemes can be unstable and often fail to learn high quality representations. however, we introduce a new training scheme which uses loss scaling and task-heterogeneous batches so that gradient steps are more evenly balanced across multiple different competing tasks, greatly improving training stability and overall performance. we call our prefinetuned models muppet; massive multi-task representation with pre-finetuning. through extensive experiments, we show that incorporating pre-finetuning to roberta @xcite and bart @xcite ) models yields consistent improvements, including new state-of-the-art performance for rte @xcite and hellaswag @xcite , without having to specify specific intermediate transfer tasks. these gains are particularly strong in the low resource regime, where there is relatively little labeled data for fine-tuning. we also study why pre-finetuning outperforms previous multitasking schemes. we first compare different optimization techniques to stabilize training, and find it important to use task-heterogeneous batches with task-rebalancing loss scaling. we also show that scale is crucial for effective multi-task learning. we empirically see a critical point in terms of the number of tasks (usually over 15); having fewer tasks degrades representations, while having more seems to improve performance linearly as far as we were able to scale. to summarize, our contributions include: ## related work multi-task learning has been an increasingly active topic in recent literature. recent advances such as mt-dnn show that by leveraging multitask learning, we can further improve performance on several language benchmarks on top of traditional pre-training @xcite . however, t5 @xcite shows that incorporating multi-task learning ontop of larger models does not improve upon the standardized pre-training / finetuning. thus the effect of multi-task learning across different pre-training methods is not fully understood. recently @xcite showed how doing mtl training on a range of qa tasks can improve the performance of t5 by taking advantage of cross dataset transfer. unlike our approach, they convert all the data to a seq2seq format, operate on a smaller mtl scale, have a different batching strategy, and focus solely on improving qa tasks. our work shows how even seemingly very different datasets, for example, summarization and extractive qa, can help each other by improving the model's representations. our work aims to explore multi-task learning at a much larger scale; by incorporating a larger number of tasks, we show that we can consistently improve several language benchmarks from several ## pre-finetuning through massive multitask learning previous work has reported mixed results from experiments on multi-task learning @xcite . in general, it can be challenging to balance the losses from different tasks; upsampling can lead to overfitting low resource tasks, and downsampling can lead to improper learning of specific tasks. this difficulty is particularly pronounced when operating at the scale of experiments we show in section 5.1, where there are more diverse tasks than previously considered. this section presents our pre-finetuning approach that leads to more stable and accurate multi-task training by introducing new optimization, loss scaling, and task sampling schemes to balance each minibatch's updates better. ## empirical results we first show that pre-finetuning improves the representations of pre-training models. to do so, we fine-tune our pre-finetuned models on a large set of tasks. for each of the individual downstream tasks, we use a fixed hyper-parameter search to optimize over simple hyperparameters such as learning rate, @xcite and dropout @xcite . we present our results in two tables. given that our pre-finetuned models now have an understanding of the task at hand through the use of classification heads, we have a choice during finetuning on whether or not to use these heads. in general we found re-using heads to be beneficial for mrc, commonsense and sentence prediction tasks with small dataset size. across the board, pre-trained representations that were further refined with pre-finetuning outperformed standard pre-trained representations. we see more modest gains on larger datasets, most likely because we do not need to refine representations beforehand if the fine-tuning dataset is large. on smaller datasets, we see substantial gains. for example, the pre-finetuned roberta-base model on rte improves by close to 9 points, rivaling the roberta-large accuracy, while the pre-finetuned roberta-large model gets new state-of-the-art on rte rivaling models an order of magnitude larger than it. we do not improve just over sentence prediction tasks but on every set of tasks that we measured. for example, we reach a new state of the art on the hellaswag dataset previously achieved by utilizing a new fine-tuning approach. our methods do not increase parameter count or any complexity measures but are quite successful at refining features and preparing them for downstream fine-tuning. ## conclusion in this work, we propose pre-finetuning, a stage after pre-training to further refine representations before end-task finetuning. we show that we can effectively learn more robust representations through multi-task learning (mtl) at scale. our mtl models outperform their vanilla pre-trained counterparts across several tasks. our analysis shows that properly scaling mtl with heterogeneous batches and loss scaling is critical to leveraging better representations. we also show a critical point regarding the number of tasks when doing multi-task learning, where fewer tasks degrade representations compared to the pre-trained model, but more tasks than this point improve representations.