E X T5: T OWARDS E XTREME M ULTI -T ASK S CALING FOR T RANSFER L EARNING
A BSTRACT
Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training.
Towards this goal, this paper introduces E X M IX ( Ex treme Mix ture): a massive collection of 107 supervised
NLP tasks across diverse domains and task-families.
Using E X M IX , we study the effect of multi-task pre-training at the largest scale to date, and analyze cotraining transfer amongst common families of tasks.
Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own.
Finally, we propose E X T5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised E X M IX .
Via extensive experiments, we show that E X T5 outperforms strong T5 baselines on SuperGLUE,
GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of E X M IX .
E X T5 also signiﬁcantly improves sample efﬁciency while pre-training.
I NTRODUCTION
Transfer learning (Schmidhuber, 1987;
Pratt et al., 1991; Caruana et al., 1995) has been the cornerstone of recent progress in natural language processing (Ruder et al., 2019; Devlin et al., 2019;
Raffel et al., 2020).
While self-supervised pre-training has been shown to be highly effective at exploiting large amounts of unlabeled data without relying on human annotation, there is still much to explore regarding transfer learning in a multi-task co-training setup.
Prior seminal works like T5 (Raffel et al., 2020) and MT-DNN (Liu et al., 2019a) have demonstrated a degree of promise in the paradigm of multi-task co-training (Caruana, 1997).
However, the challenge of catastrophic forgetting remains.
Tasks often have to be carefully selected in order to demonstrate positive afﬁnity with regards to downstream transfer.
In many cases, it is not unreasonable to expect negative transfer (Rosenstein et al., 2005; Vu et al., 2020).
This makes the process of empirically curating a set of tasks to include in a transfer learning setup both computationally prohibitive and speciﬁc to downstream tasks.
While standard pre-training typically employs a variant of the self-supervised language modeling objective (Raffel et al., 2020), certain types of skills such as commonsense knowledge are only acquired at a slow rate even using massive amounts of unlabeled data (Zhang et al., 2021).
As ever larger models are trained, the development of much more sample-efﬁcient pre-training settings becomes thus more important, and could be addressed via multi-task learning.
For the ﬁrst time, we explore and propose Extreme Multi-task Scaling — a new paradigm for multitask pre-training.
Compared to the largest prior work (Aghajanyan et al., 2021), our study doubles the number of tasks and focuses on multi-task pre-training rather than ﬁne-tuning, which enables a direct comparison to standard pre-training.
Our proposal is based on the insight that despite negative transfer being common during ﬁne-tuning, a massive and diverse collection of pre-training tasks is generally preferable to an expensive search for the best combination of pre-training tasks.
To this end, we introduce E X M IX : a massive collection of 107 supervised NLP tasks to be included in a multi-task pre-training setup.
We process all tasks in an encoder-decoder friendly format to readily support the sharing of all parameters across all tasks.
We postulate that an ensembling effect across as many tasks, distributions and domains as possible results in a consistently net-positive outcome.
This echoes early multi-task learning results (Caruana, 1997; Baxter, 2000) indicating that a bias that is learned on sufﬁciently many tasks is likely to generalize to unseen tasks drawn from the same environment.
Moreover, our experiments verify that our E X M IX mixture outperforms a best-effort mixture of manually curated tasks.
Finally, we propose E X T5: a T5 model (Raffel et al., 2020) pre-trained on a mixture of supervised
E X M IX and self-supervised C4 span denoising.
E X T5 outperforms state-of-the-art T5 models on well-established benchmarks such as SuperGLUE (Wang et al., 2019a), GEM (Gehrmann et al., 2021), and Rainbow (Lourie et al., 2021); as well as Closed-Book QA (Roberts et al., 2020) tasks.
Notably, our experimental ﬁndings also suggest that including E X M IX may reduce the number of pre-training steps required to achieve strong performance, bringing about substantial sample efﬁciency beneﬁts.
To summarize, this paper contributes the following:
– We propose E X M IX (§2): a collection of 107 supervised NLP tasks for Extreme Multi-task Scaling , formatted for encoder-decoder training.
E X M IX has approximately twice as many tasks as the largest prior study to date (Aghajanyan et al., 2021), totaling 18M labeled examples across diverse task families.
– Given this large collection of tasks, we conduct rigorous empirical studies evaluating transfer between common task families (§2.1).
Our experiments show that curating a pre-training mixture based on ﬁne-tuning transfer is not straightforward (§2.2).
Hence, efﬁciently searching for the best subset of tasks to include in a multi-task pre-training setup is challenging and prohibitive.
– Using E X M IX , we pre-train a model alongside the C4 span-denoising objective introduced by Raffel et al.
(2020), resulting in a new pre-trained model which we call E X T5 (§3).
E X T outperforms state-of-the-art T5 on well-established benchmarks such as SuperGLUE, GEM, Rainbow, Closed Book Question Answering, and several other tasks that are outside of E X M IX (§3.2), while also being more sample-efﬁcient (
T HE E X M IX T ASK C OLLECTION
[Figure]
To explore the Extreme Task Scaling paradigm, we introduce E X M IX ( Ex treme Mix ture), a collection of 107 diverse English NLP tasks totaling 18M examples.
Following Raffel et al.
(2020), we format all tasks as text-to-text examples to readily allow for multi-task training.
This uniﬁed format also enables simple implementations without the need for taskspeciﬁc heads/losses, loss scaling, or explicit gradient accumulation for heterogenous batches as in prior works (Liu et al., 2019a; Aghajanyan et al., 2021).
When selecting examples from E X M IX , examples from each dataset are sampled proportionate to the individual dataset’s size, with each dataset’s sampling rate capped at 3 × 10 5 maximum effective examples to ensure a balance between large and small datasets.
We refer readers to Appendix A for a comprehensive breakdown of
E X M IX .
Additionally, we discuss future multilingual variants of E X M IX and E X T5 in §5.
[Equation]
As discussed in §1, our goal is to pre-train a model on E X M IX to improve downstream performance.
One natural question to ask is which tasks have a negative impact on downstream performance?
Speciﬁcally, is there a subset of E X M IX that leads to better representations when used for multitask pre-training?
Obviously, testing all possible 2 | E X M IX | combinations is impractical since the pre-training process is expensive to run.
Instead, we experiment with the less expensive co-training procedure (i.e., multi-task ﬁne-tuning), using representative subsets of similar tasks.
Later, in §2.2, we explore whether these results can be used to inform the task selection for multi-task pre-training.
[Table]
To study transfer amongst task-families in E X M IX , we construct subsets (Table 1) of 3 tasks each that are partitioned along their task family.
Using these subsets, we evaluate transfer among task families in a multi-task learning (co-training) setup.
While other types of tasks are available in E X M IX , we did not include them because they were not diverse enough to be representative of a task family, and would scale the number of models needing to be trained at a quadratic rate.
Experimental Setting We ﬁne-tune a model on each pair of task families (i.e., 6 datasets at a time).
To ensure a fair balance of tasks, we sample tasks proportional within their family, but uniformly between task families.
For example, while evaluating how classiﬁcation tasks and NLI tasks transfer amongst each other, the sampling ratio of
MNLI:ANLI will be proportional (approximately 2.4:1), but the overall ratio of NLI examples to classiﬁcation examples will be 1:1.
For reference, we also train a model on each individual task family using proportional example mixing (Sanh et al., 2019).
In total, this results in F + (cid:0) F 2 (cid:1) models trained, where F is the number of task families.
Our experiments use F =as shown in Table 1, resulting in 34 models trained in total.
Each model is ﬁne-tuned on top of the released T5.1.1 BASE checkpoint for 200k steps using a batch size of 128 and a constant learning rate of 10 − 3 .
Observations We summarize our results in Table 2.
We observe that although there exist particular task-family pairs that show positive transfer (e.g., co-training with NLI helps most other tasks), negative transfer is more common when training across task families compared to intra-family training.
21 out of the 56 inter-family relationships perform worse than intra-family models with the same data budget, which grows to 38 out of 56 for a ﬁxed compute-budget.
While the abundance of negative transfer among diverse task families is an expected result, interesting trends manifest in the individual relationships.
For example, summarization tasks generally seem to hurt performance on most other task families; and CBQA tasks are highly sensitive to multi-task ﬁne-tuning.
We also report correlations for intra-family datasets in Figure 2 using the same models as in Table 2.
In most cases, we see positive correlations between datasets in the same family.
In a few cases, however, we observe an opposite trend.
For example, ﬁne-tuned models that performed better on the GEM schema-guided dialog dataset achieved lower scores on KILT Wizard-of-Wikipedia.
This initial exploratory analysis highlights both the potential of E X M IX as a tool to systematically study task relationships, as well as the potential challenges in leveraging multi-task learning naively on top of pre-trained representations.
2. C AN FINE - TUNING TASK RELATIONSHIPS HELP CURATE A PRE - TRAINING MIXTURE ?
Our observations in §2.1 showed that multi-task co-training on top of existing pre-trained checkpoints is not straightforward, and often results in negative transfer.
However, the uncovered task relationships might help efﬁciently search for an ideal subset of E X M IX for multi-task pre-training.
To this end, we select a set of the most promising task families to be included in a multi-task pretraining setup, ranking task families by the average relative improvement they provide to other target families (the last column in Table 2).
Speciﬁcally, we include NLI, commonsense, classiﬁcation, and closed-book QA tasks from E X M IX to form a mixture of 48 tasks to include in a multi-task
[Table]
[Table]
[Figure]
[Table]
[Figure]
Performance on datasets from some task families are highly correlated (e.g., NLI) whereas other task families have more erratic results across their datasets (e.g.
Dialogue pre-training setup.
We then ﬁne-tune the resulting model on SuperGLUE, comparing it to T5.1.
and E X T5 in Table 3.
While this model narrowly outperformed T5.1.1, it did not yield better results than including all of
E X M IX in the multi-task pre-training mixture, as we report in §3.2.
Moreover, it did not outperform a random selection of 55 tasks on average, as we report in our task scaling experiments (§2.5).
[Table]
the number We conclude that the negative transfer during multi-task ﬁne-tuning does not necessarily inhibit pre-training .
While we cannot directly conclude that an ideal subset of E X M IX does not exist to be mixed with self-supervised pre-training for speciﬁc downstream tasks, our experiments show that randomly including more diverse pre-training tasks generally improves downstream performance.
It must also be noted that the end-goal is to ﬁnd a mixture that leads to a general pre-trained model that can be used for a large variety of downstream tasks, and that a setup to ﬁnd a pre-training mixture tailored for
SuperGLUE would be different.
2. M ULTI - TASK P RE - TRAINING VS P RE - FINETUNING
Instead of pre-training on E X M IX , another way to leverage multi-task learning is as an intermediate step between pre-training and ﬁne-tuning.
This is referred to as pre-ﬁnetuning by Aghajanyan et al.
We conduct controlled experiments to compare pre-training with pre-ﬁnetuning.
We begin with a standard T5 base checkpoint and pre-ﬁnetune it with E X M IX .
After this phase, we ﬁne-tune on SuperGLUE.
[Table]
Table 4 compares pre-ﬁnetuning and our proposed multi-task pre-training.
We also report the total compute (in total number of tokens processed) by the model in both schemes.
The results show that multitask pre-training is signiﬁcantly superior to preﬁnetuning.
A potential hypothesis is that multi-task pre-training narrows the gap between pre-training and ﬁnetuning data distributions, as the pre-training stage more closely resembles ﬁne-tuning.
Conversely, segregating pre-training and pre-ﬁnetuning into two different stages may induce catastrophic forgetting of the pre-training task.
Hence, in
E X T5, we opt for multi-task pre-training over pre-ﬁnetuning.
2. H OW MUCH LABELED DATA SHOULD BE MIXED ?
[Figure]
without using E X M IX ( R → ∞ )
In this section, we explore how the ratio of C4 to
E X M IX examples during massive multi-task pretraining affects performance.
As mentioned later in §3, this is controlled by a hyperparameter R , where a pre-training batch will have approximately
R times as many C4 examples compared to E X M IX .
From our results in Figure 3, we ﬁnd that despite
E X M IX improving downstream performance when mixed with self-supervised C4 pre-training at many rates, a model trained with R = 0 suffers greatly in comparison.
This result is signiﬁcant, as it shows that while E X M IX improves the pre-training process, self-supervised training over a large unstructured corpus is still crucial.
2. D OES ADDING MORE TASKS HELP ? T ASK SCALING EXPERIMENTS
[Figure]
In this section, we explore how model performance changes as the number of tasks included in a massive multi-task pre-training setup is scaled up.
We choose random sets of 30, 55, and 80 tasks (each a superset of the last), pre-train a BASE -sized model for 524k steps, and ﬁne-tune them on SuperGLUE.
We train our models with batch sizes of 128 andand R = 2 (the ratio of C4 to E X M IX examples) as this conﬁguration worked best for our BASE -sized models (§2.4).
We repeat this over three random seeds (for random subset selection), and report average scores in Figure 4.
Overall, with large batches, we can see that increasing the number of tasks being mixed generally helps downstream performance.
This reinforces our intuition that task scaling indeed helps.
With small batches, there is less of an upward trend, signifying that large batches are essential for a large number of tasks.
This is intuitive, given that multi-task learning may cause gradients to be noisy (Yu et al., 2020).
Another explanation as to why this happens is that large-batch training can offer beneﬁts even for single-task models (Smith et al., 2018) — a trend formalized by McCandlish et al.
2. I MPROVING S AMPLE E FFICIENCY WITH E X M IX
[Figure]
We hypothesize that extreme multi-task scaling also improves the sample efﬁciency of pre-training.
To test this, we exclude SuperGLUE from E X M IX , pretrain a large model for 200k steps, and ﬁne-tune it on SuperGLUE at several intervals during early pretraining stages.
We ﬁnd that E X M IX pre-training is signiﬁcantly more sample-efﬁcient than vanilla selfsupervised pre-training.
Note that at only 20k pretraining steps, our ExT5 model already achieves 75.
SuperGLUE score, which outperforms a fully pretrained BERT large model by about +4% (Wang et al., 2019a).
T HE E X T5 M ODEL
To overcome the challenges of multi-task co-training at scale, i.e. negative transfer and catastrophic forgetting explored in §2.1, the rest of this paper revisits the multi-task pre-training paradigm introduced by Raffel et al.
(2020) via extreme multi-task scaling.
This section introduces E X T5: a pre-trained sequence-to-sequence Transformer encoder-decoder model (Vaswani et al., 2017) based on the popular T5 framework.
3. T RAINING E X T
Pre-training We pre-train on a mixture of C4 and E X M IX (§2), and combine them with a hyperparameter R that is the ratio at which C4 examples are sampled with respect to E X M IX examples.
The C4 objective we use is the same as that used by Raffel et al.
(2020), and
every task optimizes the standard sequence-to-sequence cross-entropy loss.
We pre-train E X T5 on the same number of steps as T5, and E X T5 sees an identical number of tokens to the released T5 models.
Concretely, we pre-train our models for 1M total steps with a batch size of 2048 and sequence length 512, resulting in a total of approximately 1T tokens seen by the model during pre-training (both unsupervised and supervised inclusive).
We use the T5.1.1 architecture (Shazeer, 2020) for all of our experiments — which uses GEGLU-activated layers instead of ReLU in classic Transformer models (Vaswani et al., 2017).
For optimization, we use Adafactor with an inverse square root learning rate schedule that kicks in after a a constant phase of 0.01 for 10k steps.
E X T5 also uses the same tokenizer as T5.
Fine-tuning We follow the same ﬁne-tuning procedure for T5 and E X T5 for fair comparison, although we found that E X T5 generally beneﬁtted from a smaller learning rate while ﬁne-tuning ( 10 −worked well for E X T5 vs 10 − 3 for T5 variants).
Fine-grained details can be found in Appendix B.
3. E XPERIMENTAL S ETUP
Our experiments consider both within-mixture and out-of-mixture tasks (i.e., whether a task is included in E X M IX ).
Within-mixture tasks measure the amount the task beneﬁts from multi-task pre-training and extreme task scaling.
Similar to the co-trained models in Raffel et al.
(2020), we continue to ﬁne-tune on the target task from a pre-trained ExT5 checkpoint.
For out-of-mixture tasks, we consider possibly new unseen tasks or collections that were not included in the E X M IX mixture to test the effect of generalizing to unseen tasks.
For the sake of brevity, the ﬁne-grained details of these experimental setups can be found in the Appendix.
3. E XPERIMENTAL R ESULTS W ITHIN - MIXTURE R ESULTS
We report results on SuperGLUE (Table 5), GEM (Table 6), Rainbow (Table 7), MsMarco (Table 8) and CBQA datasets (Table 9).
On the whole, we observe that ExT5 consistently outperforms strong T5 baselines across a range of model sizes.
On SuperGLUE, we achieve +5% , 2 . 3% and
+0 .
7% gain on BASE , LARGE and XL respectively.
On GEM, ExT5 outperforms T5 on 6 out of 9 collections while remaining on-par on the other 3 collections.
Notably, the gain on datasets such as WebNLG are approximately +11% ROUGE for the large model and generally range from +1% to +6% on different collections.
On Rainbow, ExT5 outperforms our own run of T5 by +0.7% on average and +4 .
6% improvement over the best multi-task (sequential) setup in Lourie et al.
Finally, on question answering and ranking
[Table]
ExT5 substantially outperforms T5 at two different sizes.
[Table] [Table]
[Table] [Table]
[Table]
O UT - OF - MIXTURE R ESULTS
We are also interested in evaluating E X T5 on tasks outside of E X M IX , and hypothesize that the extreme multi-task pre-training of E X T5 will lead to better performance on new unseen settings.
Concretely, we ﬁne-tune and evaluate on Machine Translation : translating sentences from English to other languages (Bojar et al., 2014; 2015; 2016);
Reasoning : answering scientiﬁc questions on
ARC (Clark et al., 2018); and Named Entity Recognition : extracting all entities from sentences on the CoNLL-2003 NER dataset (Tjong Kim Sang & De Meulder, 2003)
[Table]
Table 10 summarizes the results on the out-of-mixture tasks.
Across all tasks, we see that E X T outperforms upon T5 baselines.
The largest improvement is on the ARC scientiﬁc reasoning task, perhaps due to the large amount of QA tasks in E X M IX .
Though, the trend is consistent also with the NER and MT tasks that do not have any similar dataset in E X M IX .
This suggests that the representations learned by E X T5 are more general adaptable to a new objective, even when the output is in a new language.
This improved generalization of E X T5 is very encouraging from a practical stand point, since pretraining again with E X M IX ∪ { t } for any new target task t would be very expensive.
Instead, we see that the extreme multi-task pre-training of E X T5 already provides improved results.
Therefore, it might only be worth repeating pre-training when the collection of training datasets grows by a signiﬁcant amount (see §2.5).
R ELATED W ORK
Improving NLP models with Multi-task Learning Collobert & Weston (2008) leverage multitask learning for relatively simple tasks like Part-of-Speech tagging.
Phang et al.
(2019) use an intermediate ﬁne-tuning stage using four tasks with large datasets for Natural Language Understanding.
Similarly, Liu et al. (2019a) proposed MT-DNN, which uses a setup at a scale of around 30 tasks and up to 440M parameters.
Most recently, Aghajanyan et al.
(2021) use around 50 tasks and models of sizes upto 440M parameters.
Gururangan et al.
(2020) take an alternative approach, which is to continue pre-training a model but use domain-speciﬁc data as an intermediate step.
McCann et al.
(2018) proposed a uniﬁed framework similar to that of T5.
Recently, Wei et al.
(2021) also illustrated how a multi-task learning stage can greatly improve the zero-shot prompting performance of large language models at the scale of ~137B parameters.
Efforts have also been made to tailor pre-training objectives to speciﬁc tasks, e.g., question answering (Ram et al., 2021;
Jia et al., 2021), dialogue (Li et al., 2020), and span selection tasks (Joshi et al., 2020).
Relationships amongst different tasks Bingel & Søgaard (2017) conducted a study similar to ours in §2.1 but for more traditional NLP tasks like chunking, CCG tagging, POS tagging, etc.
More recently, Vu et al.
(2020) conducted an in-depth study of relationships between various classiﬁcation/regression, question-answering, and sequence-labeling tasks, and proposed a task-embedding framework to predict such relationships.
Khashabi et al.
(2020) also conducted similar experiments but speciﬁc to question-answering datasets/formats, resulting in a strong QA model known as UniﬁedQA that is also based on the T5 framework.
Outside of NLP, Zhang & Yeung (2010) introduced a convex optimization objective for learning task relationships, and Li et al.
(2018) explore and exploit task relationships on a variety of diverse datasets.
Choosing which tasks to transfer from Our experiments in §2.2 attempted to empirically select a set of tasks to transfer from.
Along these lines, Ruder & Plank (2017) use a Bayesian Optimization method with similarity measures to automatically select relevant data from different domains.
Similarly, Guo et al.
(2019) use multi-armed bandits to select tasks and a Gaussian Process to control the mixing rates for the selected tasks.
Another strand of recent work selects appropriate transfer languages based on manually deﬁned features (Lin et al., 2019; Sun et al., 2021).
Aside from the
NLP domain, Fifty et al.
(2021) proposed a method to select which tasks to transfer to based on task gradients.
All of the aforementioned works select data tailored to a downstream task of interest.
If a general pre-trained model was attempted to be trained in a similar fashion, computational bottlenecks similar to those motivating §2.1 and §2.2 would arise.
Pre-trained Transformers Transformer models (Vaswani et al., 2017) such as T5 (Raffel et al., 2020), BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020) rely on large unlabeled corpus for self-supervised learning.
Given the wild success of the pre-train-ﬁnetune paradigm, the search for suitable pre-training tasks has also become an active area of research (Lewis et al., 2019;
Lan et al., 2019; Chang et al., 2020; Zhang et al., 2019; Lourie et al., 2021)
While there has been evidence that supplementary pre-training tasks can help improve performance, this work is the ﬁrst massive-scale multi-task pre-trained model.
Scaling Laws Scaling laws for Transformers have attracted much attention recently, especially pertaining to model size (Kaplan et al., 2020; Zhai et al., 2021; Tay et al., 2021a).
In Kaplan et al.
(2020), the authors further investigate scaling with respect to dataset size (on the same pre-training corpus).
To this end, this work can be interpreted as an attempt of scaling up with respect to the number of high quality, diverse labeled tasks that can be used for pre-training.
E PILOGUE
Limitations Despite our best efforts to evaluate on as many representative tasks as possible while also maintaining a balance among task partitions for a given set of transfer learning experiments, any study that explicitly abstracts datasets into “task families” is highly dependent on nuances pertaining to the nature, domain, and expressiveness of the task family’s representative datasets.
For this paper, the subsets were constructed so as to include a diverse set of datasets to evaluate on, and we tried to partition task-families to be as mutually exclusive as possible.
However, it must be acknowledged that no dataset is perfectly isolated, and any set of them only a proxy for a larger “task family”.
On a separate note, lexical metrics like BLEU/ROUGE are useful but do not paint the full picture of how well a model truly performs on text-generation tasks.
Future Work We believe that a multilingual version of E X T5 would be a natural extension of this work.
Such a model will require extra care with regard to balancing not only task families, but also task languages.
A multilingual version of E X M IX could provide a more robust foundation for the analysis of task families in existing works that analyze how multilingual NLP models transfer amongst different languages (Kudugunta et al., 2019;
Hu et al., 2020; Wang et al., 2021).
For example, it would be interesting to understand whether our results in §2.1 hold across different languages (and language families), and to explore cross-lingual cross-task generalization.
We also hypothesize that modeling innovations that introduce inductive biases designed to exploit multi-task learning setups (Ha et al., 2016; Tay et al., 2021b) can push the boundary of the strong performance displayed by E X T5.
Other solutions like gradient manipulation (Yu et al., 2020; Wang et al., 2021) might also further improve extreme multi-task scaling, albeit at the cost of more complex implementations.
Conclusion This paper explores how supervised multi-task learning at a massive scale can be used to improve existing self-supervised pre-training strategies for NLP models, and does so by introducing E X M IX (§2) and E X T5 (§3).
Our experiments showed that while negative transfer is common when ﬁne-tuning on diverse tasks (§2.1), scaling up the number of tasks to include in a multi-task pre-training setup enables strong downstream performance (§3.2) with better sampleefﬁciency (§2.6).
We hope that this paper motivates future research on how existing labeled datasets can be used to further improve NLP models within the pre-train/ﬁne-tune paradigm. A CKNOWLEDGEMENTS
The authors would like to thank Mostafa Dehghani and Adhi Kuncoro for valuable comments and insights.
We would also like to thank the authors of Mesh Tensorﬂow (Shazeer et al., 2018) and T (Raffel et al., 2020), as their high-quality code and paper enabled this work.
A UTHOR C ONTRIBUTIONS
Vamsi co-led the project and was primarily responsible for the design, implementation, and engineering effort behind it.
Vamsi proposed and ran key experiments including (but not limited to): task-family transfer analysis, early proof-of-concepts for extreme task-scaling and E X T5, taskscaling analysis, etc.
Vamsi also wrote most of the paper, and was (jointly) responsible for the overall framing of the paper.
Yi served as the co-lead of the paper and proposed the initial idea.
Yi was responsible for most of the downstream E X T5 experiments, including SuperGLUE, Rainbow, CBQA, and Machine translation, along with large-scale pre-training of E X T5.
Yi also wrote large portions of the paper.
Tal was (jointly) responsible for the overall framing of the paper, and wrote large portions of it.
Tal contributed the Vitamin C task to the E X M IX mixture, along with running out-of-mixture experiments for Named Entity Recognition.
Jinfeng contributed the GEM benchmark to our mixture and was heavily involved in running experiments.
Steven contributed the DialoGLUE tasks to E X M IX , helped run a substantial number of experiments and contributed substantially to discussions around the paper’s framing.
Sanket was responsible for and ran experiments on GEM, and contributed the Yahoo Answers and
Argument mining tasks.
Honglei contributed the MsMarco task to E X M IX and helped with benchmarking ExT5 on MsMarco.
Honglei also contributed scripts and pipelines for obtaining reranking results on MsMarco.
Vinh contributed NewsQuiz, AgreeSum, TweetQa and TweetEval to E X M IX and contributed substantially to paper writing and the framing of the paper.
Dara contributed GPT DeepFake tasks and low-resource tasks (which were not used in the end).
Jianmo contributed the DocNLI task and helped with running experiments for out-of-mixture tasks.
Jai contributed the Race and MultiRC
Eraser tasks to the E X M IX and helped edit the paper.
Kai contributed several retrieval tasks to E X M IX , which were not included eventually.
Sebastian helped substantially with the paper narrative, writing of the paper and brainstorming.
Donald (along with Yi and Vamsi) was heavily involved in framing the early vision of the project.
Donald was also deeply involved in brainstorming sessions, and provided critical feedback that helped to steer the project in the right direction.
All authors contributed to brainstorming and discussion sessions.
E THICS S TATEMENT
Large language models have been shown to capture certain biases about the data they have been pre-trained on (Bender et al., 2020).
While a comprehensive analysis of such biases is outside of the scope of this work, it is a compelling direction to investigate to what extent the inclusion of supervised data during pre-training can help mitigate such biases.
An alternative consideration is the addition of diverse values-targeted data (Solaiman & Dennison, 2021) during pre-training in order to instill beneﬁcial biases in a model.
Another challenge when training large models is their energy consumption and environmental impact (Strubell et al., 2019).
To ablate different task combinations, we performed experiments using the more computationally efﬁcient ﬁne-tuning setup.
We have shown that E X M IX leads to more sample-efﬁcient pre-training compared to standard self-supervision, which we hope will save compute in future experiments.
R EPRODUCABILITY S TATEMENT
All of the modeling and training code used for E X T5 and its variants is already open-sourced as a part of the Mesh Tensorﬂow 1 (Shazeer et al., 2018) and T5 2 (Raffel et al., 2020) Libraries.
Additionally, E X M IX is composed of datasets that are already publicly available.
A D ATASETS
[Table]
Description
No.
Train Datasets |D|
Citation GLUE
General Language Understanding 949,
Wang et al.
(2019b)
[Table]
General Language Understanding 185,
Wang et al. (2019a)
KILT
Knowledge-Intensive Language Tasks3,129,
Petroni et al.
Rainbow
Commonsense Reasoning 324,
Lourie et al.
GEM (en)
Natural Language Generation 1,067,
Gehrmann et al.
DialoGLUE
Dialogue Understanding 76,
Mehri et al.
(2020) TweetEval
Twitter Classiﬁcation Benchmark 120,
Barbieri et al.
(2020) CNN/Dailymail
News Summarization 287,
See et al.
XSum
News Summarization 203,
Narayan et al.
Multi-News
News Summarization 44,
Fabbri et al.
AESLC
Email Summarization 14,
Zhang & Tetreault (2019) Gigaword
Summarization 3,803,
Rush et al.
SamSum
Dialogue Summarization 14,
Gliwa et al.
ANLI
Adverserial NLI 162,
Nie et al.
ESNLI
Explainable NLI 549,
DeYoung et al.
AgreeSum Entailment
Article-Summary NLI 7,
Pang et al.
(2021) DocNLI
Document NLI 942,
Yin et al.
Vitamin C
Fact-checking NLI 370,
Schuster et al.
(2021) Web Questions
QA (open)
Berant et al.
(2013) SQuAD
QA (context) 87,
Rajpurkar et al
[Table]
Table 11 summarizes the 107 datasets included in E X M IX .
Some of the lines in the table represent existing benchmarks that group several tasks together.
From each collection, we use the datasets that include English training data:
– GLUE : CoLA (Warstadt et al., 2019), SST-2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), QQP, STS-B (Cer et al., 2017), MNLI (Williams et al., 2018), QNLI (Converted from Rajpurkar et al.
, RTE (Dagan et al., 2006), WNLI (Sakaguchi et al., 2020).
– SuperGLUE : BoolQ (Clark et al., 2019), CB (De Marneffe et al., 2019), COPA (Roemmele et al., 2011), MultiRC (Khashabi et al., 2018), ReCoRD (Zhang et al., 2018), RTE (Dagan et al., 2006), WiC (Pilehvar & Camacho-Collados, 2019), WSC (Levesque et al., 2011).
– KILT : FEVER (Thorne et al., 2018), AIDA (Yosef et al., 2011), WNED (Guo & Barbosa, 2014), T-REx (Guo & Barbosa, 2014), NQ (Kwiatkowski et al., 2019), HoPo (Yang et al., 2018), TQA (Joshi et al., 2017), ELI5 (Fan et al., 2019), WoW (Dinan et al., 2019).
– Rainbow : α NLI (Bhagavatula et al., 2020), CosmosQA (Huang et al., 2019), HellaSWAG (Zellers et al., 2019), PIQA (Bisk et al., 2020), SocialIQA (Sap et al., 2019), WinoGrande (Sakaguchi et al., 2021).
– GEM (en) :
Wiki-Lingua (Faisal Ladhak & McKeown, 2020), WenNLG (Gardent et al., 2017; Castro Ferreira et al., 2020), CommonGEN (Lin et al., 2020), E2E (Duˇsek et al., 2019), DART
(Radev et al., 2020), ToTTo (Parikh et al., 2020), Wiki-Auto (Jiang et al., 2020), TurkCorpus(Xu et al., 2016) – DialoGLUE : Banking77 (Casanueva et al., 2020), HWU64 (Liu et al., 2019b), CLINC (Larson et al., 2019), SGD (Rastogi et al., 2020), TOP (Gupta et al., 2018).
– TweetEval : Emotion Recognition (Mohammad et al., 2018), Emoji Prediction (Barbieri et al., 2018), Irony Detection (Van Hee et al., 2018), Hate Speech Detection (Basile et al., 2019), Offensive Language Identiﬁcation (Zampieri et al., 2019), Sentiment Analysis (Rosenthal et al., 2017), Stance Detection (Mohammad et al., 2016).
B E XPERIMENTAL D ETAILS
This section describes the experimental details
B. I MPLEMENTATION D ETAILS
Our models were trained using Mesh Tensorﬂow (Shazeer et al., 2018) using the T5 library (Raffel et al., 2020).
B. D ATASET E XPERIMENTAL S ETUP
This section reports the dataset and experimental setup on each individual target task/dataset.
SuperGLUE We ﬁnetune on the entire SuperGLUE as a mixture with proportionate sampling in similar fashion to (Raffel et al., 2020).
We ﬁnetune for a total of 200k steps with a batch size of 128 .
When selecting checkpoints on SuperGLUE, we follow the same convention as Raffel et al.
(2020) in selecting the best checkpoint for each task for a fair comparison to models that are ﬁne-tuned on the individual tasks instead of co-training on all of them.
GEM We report test set results on all datasets except CommonGen and ToTTo, on which we report validation scores.
We sweep over learning rates of 10 − 3 , 5 × 10 − 4 and 10 − 4 .
All results are computed using GEM-metrics 3 .
For each dataset, we select the best model checkpoint using average of BLEU, ROUGE-1, ROUGE-2 and ROUGE-L scores on the validation set.
We use the greedy decoding strategy to be consistent with the original GEM paper (Gehrmann et al., 2021).
CBQA We report validation set results, and sweep over learning rates of 10 − 3 and 10 − 4 .
Rainbow We multi-task co-train on all datasets, and sweep over learning rates of 10 − 3 and 10 − 4 .
WMT Machine Translation We ﬁnetune our models on three collections of WMT, namely EnDe,
EnFr and EnRo.
We use a constant learning rate of 10 − 3 and dropout of 0 .
We train with a batch size of 4096 for a maximum of 400 k steps and report peak validation BLEU score.
We use a beam size of 4 and a length penalty of 0 . 6 .
ARC
We report scores on the Challenge set, and train with a batch size of 32 and sweep over learning rates of 10 − 3 and 10 − 4 .
CoNLL-03 NER We convert NER to seq2seq by writing the target as the ordered sequence of tags and entities (for example “When Alice visited New York” → “[PER] Alice [LOC] New York” ).
Accuracy is measured on a sentence level, considering a prediction to be correct only if it exactly matches the reference sequence.
C D ETAILED E XPERIMENTAL R ESULTS
Many of our experiments in §2 used the average SuperGLUE score of a model for evaluation.
We report the full results on all datasets below.
[Table]
[Table] [Table]
[Table] [Table]
[Table]
80 (random) 80.
Batch Size = 82.
30 (random) 80.
[Table]
[Table] [Table]