N OT A LL T ASKS ARE B ORN E QUAL : U NDERSTANDING Z ERO -S HOT G ENERALIZATION
A BSTRACT
Recent work has achieved remarkable zero-shot performance with multi-task prompted pretraining, but little has been understood.
For the first time, we show that training on a small number of key tasks beats using all the training tasks, while removing these key tasks substantially hurts performance.
We also find that these key tasks are mostly question answering (QA) tasks.
These novel findings combined deepen our understanding about zero-shot generalization—training on certain tasks such as QA encodes general knowledge transferable to a wide range of tasks.
In addition, to automate this procedure, we devise a method to identify and upsample key training tasks without observing the test tasks based on cross validation.
Empirically, our approach achieves improved results across various model scales and tasks.
I NTRODUCTION
Recent work (Brown et al., 2020; Artetxe et al., 2022; Rae et al., 2021) has demonstrated the potential of leveraging pretrained language models (PLMs) to perform zero-shot generalization.
Zero-shot generalization enables PLMs to adapt to a variety of natural language processing (NLP) tasks without relying on any annotated data, which opens the possibility towards generic systems.
Pretrained models, such as GPT-3 (Brown et al., 2020), BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020), can perform zero-shot inference on unseen test tasks by leveraging natural language prompts and formulating NLP tasks into language modeling tasks.
More recent advances (Wei et al., 2022; Sanh et al., 2022) performed multi-task prompted training on PLMs and further enhanced the zero-shot performance to a large extent.
Despite the substantial progress, few works have studied how multi-task prompted training boosts the zero-shot performance.
The lack of understanding hinders further improvement of the field.
To this end, we take a further step to understand multi-task training for zero-shot generalization (1) by selecting only a small number of key training tasks and performing multi-task training and (2) by studying the characteristics of tasks with general transfer ability.
The results reveal several interesting findings — First of all, only a small number of training tasks dominate the performance of zero-shot generalization.
In other words, using only these key training tasks to perform multi-task training leads to good results, while removing these key tasks would drastically hurt the zero-shot performance.
Secondly, not all tasks are born equal, and some tasks show general transfer ability by providing widely useful knowledge.
Moreover, key tasks with general transfer ability can be automatically detected using pairwise generalization results.
In addition, based on the findings, we propose an improved method, task resampling , which improves multi-task training for zero-shot generalization to a large extent.
Task resampling first automatically identifies a set of key training tasks based on cross-validation without peeking forward any test tasks, and then performs resampling by upsampling key tasks or downsampling non-critical tasks, as shown in Figure 3.
In this way, we build a better mixture of multi-task training sets to highlight those key tasks with better general transfer ability.
Experiments show that task resampling consistently outperforms the previous approach T0 (Sanh et al., 2022) across three different model scales and on test tasks of various types.
To sum up, our contributions are as follows.
We conduct experiments to understand and reveal how multi-task training for zero-shot generalization works—(1)
Only a small number of training tasks dominate zero-shot generalization; (2) Some key tasks provide general transfer ability and can be detected using pairwise generalization results.
We devise a novel method, task resampling, to improve zero-shot generalization by (1) first automatically identifying key training tasks based on cross validation without observing any test tasks, and (2) resampling training tasks using upsampling and downsampling strategies.
Experiments show that our approach achieves new state-of-the-art results across various model scales and tasks.
R ELATED W ORK 2. Z ERO -S HOT L EARNING IN NLP
Zero-Shot Learning denotes the setting when no data correlated with the test set is available during the training stage.
The early definition of zero-shot learning referred to predicting samples with unseen classes, so traditional methods require prior information such as semantic knowledge (Zhang et al., 2019a) or knowledge graph (Chen et al., 2021) for an unseen class so that model can predict that class without training data.
Meta-learning (Zhang et al., 2022) and reinforcement learning (Ye et al., 2020) methods are also used for zero-shot learning.
Recently, more work focuses on the setting of predicting samples with unseen tasks, supported by the development and prevalence of pre-trained language models (PLMs), as well as multi-task training.
McCann et al.
(2018) unifies NLP tasks into QA-format to perform multi-task learning.
Liu et al.
(2019) designs a multi-task deep neural network for natural language understanding tasks.
Aghajanyan et al.
(2021) designs an intermediate training stage between pretraining and finetuning using around 50 tasks.
Most recently, with the combination of the above two approaches, T0 and FLAN (Sanh et al., 2022; Wei et al., 2022) have shown that explicit multi-task prompted training where all tasks are unified by the natural language prompts can vastly promote zero-shot task generalization.
We build upon previous work within this new paradigm and devote ourselves to improving the recipe of multi-task prompted training by revealing the mechanism of generalization in Section 3 and further enhancing its performance in Section 4.
2. T HE I NTERPRETATION OF P ROMPTED L EARNING
Recent works have shown an increased interest in how the prompts help the model generalize to unseen tasks.
Some researchers (Wei et al., 2022; Schick & Sch¨utze, 2021; Mishra et al., 2022) suggest that the model learns to understand what they are doing through prompts.
While some work (Webson & Pavlick, 2022; Logan IV et al., 2022) challenges this assumption, revealing that sometimes we could get comparable performance without prompts or even with wrong prompts.
T0 (Sanh et al., 2022) claims that they only empirically witness the transfer phenomenon, but it is unclear why it happens.
We provide new insights into the reason for generalization.
We challenge the idea that the model learns the task through instructions, based on the observation that deleting a small but important set of tasks will lead to transfer failure.
We suggest that most generalization ability comes from key tasks, which could be divided into specific and general transfer abilities.
We hope our discovery could promote the development of this field.
2. T RANSFER R ELATIONSHIPS IN M ULTI -T ASK L EARNING
Based on the observation that transfer ability comes from key tasks, we would like to further improve the zero-shot performance by exploring the transfer relationships in multi-task learning.
Some previous works learn transfer relationships in supervised multi-task training.
Taskonomy (Zamir et al., 2018) builds a transfer structure of a series of computer vision tasks by training task-specific encoders on each dataset and retraining the decoders on target datasets.
Dwivedi & Roig (2019) and Song et al.
(2019) obtain the transfer relationship based on the assumption that transferable task-specific models have similar representations or embeddings on the same data.
Vu et al.
(2020) learns the embedding of each NLP task and tries to predict the transfer relationship between different datasets.
ExT (Aribandi et al., 2022) learns the transferability by co-training the source task and target task and evaluate on the target task.
UnifiedQA (Khashabi et al., 2020) explores the transferability between different QA tasks.
The major challenge lies in that we could not get access to the test task at the training stage, thus it is hard to figure out in advance which training tasks can be more useful for unseen tasks.
To address this challenge, we propose a reweighting method based on the transfer performance of pairwise training tasks, which will be discussed in Section 4.
2. D ATA A UGMENTATION IN NLP
Data Augmentation is widely used in NLP to strengthen the robustness and diversity of the data distribution and promote the model performance.
However, most of the approaches are conducted in word-level (Zhang et al., 2015; Wang & Yang, 2015; Wei & Zou, 2019) and sentence-level (Kafle et al., 2017; Hou et al., 2018;
Khashabi et al., 2018; Zhang et al., 2018), or in other words, instancelevel.
In our paper, to naturally cater for the multi-task setting, we adopt a new perspective and design a cross-domain data augmentation which intersects each domain data with different kinds of tasks, which significantly improves the diversity of data distribution.
U NDERSTANDING Z ERO -S HOT T ASK G ENERALIZATION
This section explores how multi-task training contributes to zero-shot generalization.
By revealing the mechanism of task transfer, we provide some insights for improving zero-shot performance.
3. D ATA
We followed the setting of T0 (Sanh et al., 2022) and adopted the tasks therein.
There are 38 training tasks across 8 task types, and 11 test tasks ranging from natural language inference (RTE (Candela et al., 2006),
CB (De Marneffe et al., 2019), ANLI/R1-R3 (Nie et al., 2020)), coreference resolution (WSC (Levesque et al., 2012), Winogrande (Sakaguchi et al., 2020)), sentence completion (COPA (Roemmele et al., 2011),
StoryCloze (Mostafazadeh et al., 2017), Hellaswag (Zellers et al., 2019)), to word disambiguation (WiC (Pilehvar & Camacho-Collados, 2019)).
Both training and test sets are disjoint in task types, thus guaranteeing the zero-shot setting.
We report the mean and median accuracy over multiple prompts for each test task.
3. A S MALL N UMBER OF K EY T ASKS D OMINATE P ERFORMANCE
Since there has been little agreement on where the excellent zero-shot generalization performance comes from, we would like to conduct experiments to explore its mechanism.
Some researchers believe that the model understands the prompts through multi-task training (Wei et al., 2022; Schick & Sch¨utze, 2021; Mishra et al., 2022), while we take an orthogonal perspective and hypothesize that there might be a few key tasks that are crucial for the zero-shot generalization performance.
For a straightforward verification of the above hypothesis, we conduct two experiments.
Single Task Shows Zero-Shot Transfer Ability We set up an experiment to study the pairwise transfer results between all pairs of tasks.
Specifically, for any pair of tasks in the T0 collection, we train on one task and evaluate on the other.
We expect to decouple the effects of multi-task learning and observe the transfer ability of single tasks.
Results are shown in Figure 1.
On the held-out test tasks, the performance gap between multi-task training and single-task training is less than 5 points on average.
We also conduct experiments by training on top-3 tasks for each test dataset.
The detailed results are presented in Appendix D.1.
A Small Set of Tasks Dominates Performance on the Test Tasks We manually select the toptasks out of all training tasks of T0.
These selected tasks empirically demonstrate good generalization to the test tasks according to preliminary experiments.
Specifically, we first select the top-3 key tasks for each test task.
Then, we select the tasks which appear at least twice in the top-3 support tasks.
As a
[Table]
M R P C QQ P P
A W S
H o t p o t
Q A
W i
k
i Q
A
A d v . /
D B i d a f
A d v . /
D B
E R T
A d v . /
D R o b e r T a
D u
o R C / S e l f .
D u o R C / P a r a .
R O P E S Q u o r e f C
o s _ E
C o s m o s
Q A
D R E
A M Q A S C Q u A I L Q u a R e l Q u a R T z
S o c i a l I Q
A
W i
k
i H
o p W
i
Q
A A m a z
o n
A pp R e v
i
e w s I M D B
R o tt e n
T o m a t o e s Y e l p
C o mm o n G e n W
i
k
i B i o
C NN D a il y
M a il
G i g a w o r d
M u l t i N e w s
S a m S u m X S u
m A G N e w s D B P
e
d i a T R E C R T E C B R 1 R 2 R 3 C
O P A
H e ll a s w a g
S t o r y C l o z e
W i n o g r a nd e W S C W
i
C
[Figure]
[Table]
[Table]
result, the number of selected key tasks is exactly 8.
The top-8 selected tasks are CosmosQA
(Huang et al., 2019), Social IQA (Sap et al., 2019), PAWS (Zhang et al., 2019b), QuAIL (Rogers et al., 2020), WikiQA (Yang et al., 2015), QuaRTz (Tafjord et al., 2019), QASC (Khot et al., 2020), and
ROPES (Lin et al., 2019).
For comparison, we experimented with three different variants by training a T5-Large model using only top-8 tasks (“Top-8 Only”) and all but the top-8 tasks (“T0 Tasks w/o Top-8”), respectively.
We also experimented with backbones of different scales (XL) and architecture (decoder-only model), and results are presented in Appendix C. [Figure]
Results on T5-Large are shown in Table 1.
We observe that training with only the top-8 tasks outperforms training with all tasks when tested on 11 test tasks, while training with all but the toptasks drastically decreases performance.
It proves that a few key tasks contribute to zero-shot task generalization.
Training with key tasks selected by post-hoc results achieves much better zero-shot performance than training with all tasks.
It is generally agreed that training with more tasks should lead to better learning of prompts, but our experiments show that this is not the key reason for the performance improvement.
The information contained in the key tasks plays a significant role in improving the few-shot generalization.
This result raises a further question whether the model learns to read, understand and react to instructions broadly from all tasks, or just benefit from several key tasks which have strong general transferability.
3. G ENERAL T RANSFER AND S PECIFIC T RANSFER
First of all, we divide the transfer ability into specific transfer ability and general transfer ability according to the scope of target tasks.
Specific transfer ability means that the task can only provide special knowledge for a small set of tasks with certain kinds of patterns.
A typical example is the sentiment analysis task, which helps the sentiment analysis tasks of different domains a lot, but has little effect on other complex NLU tasks.
The specific transfer ability is relatively stronger among the tasks at the diagonal blocks in Figure 1.
General transfer ability means that the task can provide knowledge that is required by most downstream tasks.
The more it provides beyond the knowledge captured by the pretrained model, the better it will contribute to the overall transfer ability of the model.
For example, adding question answering (QA) tasks will improve the performance on most of the downstream tasks, which could be because they provide valuable commonsense knowledge and reasoning skills.
After introducing the division of transfer ability, we raise two questions based on the experimental results in Section 3.2.
(1) Do the key tasks embrace the ability of general transfer or specific transfer?
(2) Can we reveal the common patterns shared by the tasks with general transfer ability?
3. N OT A LL T ASKS ARE B ORN E QUAL
Some QA Tasks Show General Transfer Ability For the first question, from Figure 1, most of the tasks selected in post-hoc experiments bring improvements on a wide range of tasks, and thus have a certain degree of general transfer ability.
In addition, most of the tasks that show general transfer ability are QA tasks.
Two notable concepts are QA format and QA tasks. QA tasks indeed take the QA format.
However,
QA-formatted data are not necessarily QA tasks (e.g., prompted sentiment analysis data taking the
QA format is not a QA task.).
Here, QA tasks refer to those tasks that require reasoning skills, such as reading comprehension. We conduct an experiment by formatting the sentiment analysis task into a multiple-choice QA task. Specifically, we convert the format of a sentiment analysis task:
Yelp Review Full into the multiple choice QA format (see Figure 2). We compare the zeroshot performance on 11 unseen tasks between training on raw Yelp Review Full and training on QA-formatted Yelp Review Full, which is displayed in Table 2.
Results show that simply using QA-formatted non-QA-tasks does not benefit zero-shot performance, proving that it is not simply the QA format that results in the zero-
[Table]
Considering that QA tasks account for a large portion of the T0 benchmark, which might affect our conclusion, we also validated it in another setting (i.e., QA tasks only account for 7/43 of all training tasks) and presented the results in Appendix C.3.
The results verify the effectiveness of QA tasks. Why QA Tasks Show General Transfer Ability? Experiments show that some QA tasks demonstrate general transfer ability, and we would like to explore the underlying reasons.
We suspect some QA tasks provide some knowledge that is not captured in the pretraining process.
From the examples in Table 3, we can see that both CosmosQA and SocialIQA require some simple reasoning ability in the general domain, which is required for a wide range of NLP tasks.
More importantly, it is difficult to learn this knowledge in the pretraining stage, so an additional supplement is necessary to make the model have good cognitive ability.
As a result, those tasks show better general transfer ability.
There may be some quantitative methods to evaluate the knowledge provided by these tasks.
A possible solution is to design some probe tasks, as is done in Pruksachatkun et al.
and we leave this for future work.
[Table]
Which Kinds of QA Tasks Work?
Another important observation is that not all QA tasks show general transfer ability.
Specifically, CosmosQA, SocialIQA
[Table]
and QuAIL show outstanding transfer ability, while some tasks such as WikiHop, and WiQA do not.
Through a careful examination of the datasets (see Table 3 for reference), we conjecture that there are two reasons.
First of all, the domain type matters a lot.
Taking an extreme case as an example, training on datasets full of math problems is not likely to provide general transfer ability to other tasks.
All of CosmosQA, SocialIQA, and QuAIL require commonsense knowledge that is useful in the general domain.
However, the WikiHop dataset urges the model to remember specific knowledge that is mainly required for knowledge contests.
Another possible factor is the text format.
In detail, the expressions of WikiHop/WiQA seem much more artificially-constructed than SocialIQA/CosmosQA, thus showcase limited transfer ability.
So far, we have analyzed phenomena about the zero-shot task generalization ability.
At the same time, three problems remain unsolved: (1) The test set cannot be seen in advance (2) We need to further distinguish which QA tasks are useful when a large number of QA tasks are provided (3) When the training set provided is changed, we need to distinguish new tasks with general transfer ability.
For the above three reasons, we propose a general data-driven approach to identify tasks with general transfer capability.
A N I MPROVED M ETHOD : T ASK R ESAMPLING 4. M ETHOD
We have arrived at several findings in Section 3, revealing that (1) only a small number of training tasks dominate the performance of zero-shot generalization and (2) some key tasks with general transfer ability can be detected through pairwise generalization.
Following the findings, we hypothesize that one of the key aspects of improving zero-shot performance is to appropriately adjust the weight of different training tasks, such that the model is trained with an optimized mixture of multi-task datasets.
To this end, we devise a novel method, task resampling, that first automatically identifies a set of key training tasks based on cross validation without observing any test tasks, and then re-weights the training tasks through upsampling or downsampling.
Figure 3 shows an overview of our method.
Formally, we are given a set of training tasks T
=
{ t i } i where t i is a task, and a pretrained model
M .
Each task is formulated as t i
= { x i,j , y i,j } j , consisting of a prompted input x i,j and a prompted target y
i,j .
Our goal is to assign appropriate weights { w i } i for each training task, and use the optimized mixture of training tasks to train the model M in a multi-task manner such that it performs well on unseen test tasks.
Generally, our method consists of three major steps.
Pre-detection of key tasks .
Identify key training tasks based on cross validation without relying on any test tasks.
This can be viewed as a prior approximation to the post-hoc method in Section 3.2. 2.
Task resampling .
Resample different training tasks by upsampling key tasks or downsampling non-key tasks.
Multi-task training .
Train a model using the resampled mixture of multi-task datasets.
Pre-Detection of Key Tasks Without observing the test tasks, the main idea of our approach is to use cross validation within the training tasks.
To identify the key tasks, we first train a model on each training task and evaluate it on all the training tasks.
This results in a N × N ( N is the number of training tasks) table, which is part of the results in Figure 1.
Then we design a method to select key training tasks based on this N × N table.
For each task pair A and B , let f ( A, B ) be the performance of training on A and evaluating on B .
We let g ( A, B ) =
if the following conditions are satisfied (and otherwise g ( A, B ) = 0 ):
A and B are of different task types.
The performance f ( A, B ) is high enough: f ( A, B ) ≥ max A ′ ̸
=
B f ( A ′ , B ) − THand f
( A, B ) ≥ mean A ′ ̸
=
B f ( A ′ , B ) + TH
Here TH 1 and TH 2 are two constants which control the tolerant distance between f(A, B) and maximum performance and average performance respectively.
The value g ( A, B ) indicates whether
A is a high-performing training task for B .
We constrain that A and B are of different types because we eventually target cross task generalization.
We then aggregate g ( A )
=
(cid:80) B g ( A, B ) to represent how many times A is a high-performing training task for another task, and use the tasks with the largest g ( A ) values as the key tasks.
In our implementation, we apply thresholding on g ( A ) to obtain the set of key tasks.
Task resampling by Upsampling or Downsampling After detecting the key tasks using only the training tasks, we propose a simple yet effective resampling approach to optimize the mixture of multi-task data.
We either perform upsampling or downsampling strategy.
For upsampling, we upsample the key tasks by N u times.
For downsampling, we cap the number of samples to be N d for each non-key tasks and use the original sample size for the key tasks.
In our preliminary experiments, we found task resampling more robust than using the key tasks only because it takes a softer approach to highlight the importance of key tasks while maintaining knowledge from other tasks.
Data Augmentation In the above sections, we have discussed an important discovery that certain tasks are crucial for zero-shot performance.
However, some of these key tasks might be limited in terms of labeled data.
Thus, we further propose a data augmentation method to create as many samples as possible for each task.
Specifically, given tasks A and B , we apply the prompts of A to the data of B to obtain additional augmented data.
In other words, we have more data from task
B that are used to perform the task A .
We use a trained T0 to predict the labels of the augmented samples, which is similar to self training.
Note that data augmentation is optional and independent of task resampling.
We do ablation studies to investigate the effectiveness of this component.
Multi-
Task Training Our training procedure is the same as T0.
The only difference is that we employ an optimized mixture of datasets (and optionally with data augmentation).
4. E XPERIMENTAL S ETUP
Following the same training and evaluation setting as T0 (Sanh et al., 2022), we finetune the T5LM-Adapt model on 38 training tasks, which has been discussed in detail in Section 3.1.
For data preprocessing, following T0, to balance the number of data for different tasks, we restrict the maximum data examples for each training task to be 500,000.
Based on our resampling strategy, we set the pre-detection parameters as TH 1 = 5 , TH 2 = 10 , and then choose the datasets which are counted as the key tasks at least twice (i.e., all tasks A with g ( A ) ≥ 2 ).
Given each key task D with data size | D | , we duplicate D by 5 times ( N u = 5 ) for the upsampling strategy and empirically start from 50,000 samples for each dataset.
For the downsampling strategy, we downsample each non-key task to N d = min(50 , 000 , | D | ) samples.
We provide detailed statistics about the datasets in Appendix B.1.
4. R ESULTS
Post-hoc v.s. Prior Detection of Key Tasks One vital part of our approach is the detection of key tasks, so we list the key tasks selected by post-hoc (i.e., observing the test tasks) and pre-detection methods in Table 4.
There are five common tasks shared by the two methods, indicating that our approach can detect most of the key tasks.
Moreover, even though the two sets of key tasks are not exactly matched, our experiments demonstrate that this does not affect performance.
[Table]
Main Results We experiment with three scales: Large, XL, and XXL.
We compare our proposed method with the original T0 reported in (Sanh et al., 2022) (T0 ( † )) and our reproduced T0 (T (*)).
Also, we present the results of task-resampled T0, including T0 with upsampling key tasks (US-T0) and T0 with downsampling non-key tasks (US-T0).
Besides, we display the performance of the task-resampled T0 with augmented data, dubbed as US+DA-T0, DS+DA-T0 respectively. We summarize the following key observations from Table 5.
[Table]
T0 at three different scales.
Results with † are reported by Sanh et al., and results with ⋆ are reproduced in our experiments.
US-T means T0 with upsampling key tasks, DS-T0 means T0 with downsampling non-key tasks, and DS+DA-T0 / US+DA-T0 represents DS-T0 / US-T0 with augmented data. “Our Best” is achieved with the US+DA-T0 setup.
Advantage of Task Resampling.
Our task-resampled T0 with both upsampling and downsampling strategies (US-T0 and DS-T0) boosts the performance of reproduced T0.
Specifically, DS-T0 outperforms T0 (*) by 3.2% at Large scale and 2.0% at XL scale, and US-T0 outperforms T0 (*) by 3.9% at Large scale and 0.6% at XL scale.
Advantage of Data Augmentation.
Task-resampled T0 achieves better performance with augmented data.
Specifically, downsampling T0 (DS-T0) increases 1.3% with augmented data at Large scale and upsampling T0 (US-T0) increases 1.3% with augmented data at XL scale.
And T0 with both task resampling and augmented data achieves 0.8% gain with reproduced T0.
It indicates that data augmentation can further strengthen the mixture of multi-task data.
Advantage of Our Implementation Framework.
Our reproduced T0 result is better than the reported T0 (Sanh et al., 2022) by 9.4% at XL scale and 3.2% at XXL scale.
C ONCLUSIONS
This work studies the principles of zero-shot generalization through pairwise experiments, and reveals that a small number of training tasks dominate performance.
We further divide the transfer relationship into specific transfer and general transfer, and find that adding those tasks with general transfer ability will contribute to the performance gain for most tasks.
Moreover, those tasks with general transfer ability can be identified by examining the pairwise generalization results.
Based on the findings, we propose the task resampling method to improve the zero-shot performance.
Extensive experiments demonstrate the effectiveness of our framework.
A CKNOWLEDGMENTS
Jian Li and Jing Zhou are supported in part by the National Natural Science Foundation of China Grant 62161146004, Turing AI Institute of Nanjing and Xi’an Institute for Interdisciplinary Information
Core Technology.
A E XPERIMENTAL D ETAILS A. H YPER - PARAMETERS S ELECTION
For all the experiments we adopt the ADAM optimizer and use a learning rate of 1e-4.
Considering the amount of data, the batch size and training steps are different for different settings in our experiments.
We don’t search the training hyper-parameters anymore because we find that the performance is similar as long as we train for sufficient epochs in our primary experiments.
We use different batch sizes and training steps for different amounts of data mostly for time-saving.
The hyperparameters of our experiments are in Table 6.
[Table]
For other hyper-parameters we selected, we predefine them using some primary experiments on
T5-Large, and then apply them directly on larger models for sake of time.
We first select the key tasks by searching TH 1 , TH 2 using the upsampling strategy with N u = 5 , the whole search space is
TH 1 = { 5 } , TH 2 = { 5 , 10 } .
We choose G ( A ) ≥ 2 because if you draw the distribution of G(A) values, you can clearly see that the tasks with G ( A ) ≤ 1 are the long-tailed part.
When the key tasks are selected, we then search the hyper-parameters for upsampling and downsampling using the search space N u
= { 2 , 5 } , N d = { 5 , 10 } , and choose the best one on T5-Large.
We find that those hyper-parameters don’t affect the results a lot, i.e., the gap among them is much smaller than the gap between them and the baseline.
We only report the best result on T5-XXL, because we are unable to run all the experiments due to the limitation of computing resources.
Our best result is achieved using US+DA-T0.
A. D ATA A UGMENTATION D ETAILS
We propose two algorithms for the domain-task intersection.
The first one is based on a human-written taxonomy tree, and another relies on universal fields.
We combine the two techniques and achieve a balance between quality and diversity.
Taxonomy Tree based Domain-Task Intersection To build a general taxonomy tree to cover as many prompts as possible, we take both task format and task content into consideration to develop a series of guidelines similar to a Decision Tree.
Then, at the end of each branch, we intersect the source data lying in that branch with the related prompts belonging to that branch.
In this way, we might produce a reasonable combination of the source data and prompt.
For example, classification tasks like IMDB can also do tasks like title generation.
Universal Domain-Task Generation To further improve the diversity of the augmented data, we get rid of the man-made restriction and propose universal domain-task generation.
In detail, we define the unified fields for data from all domains, which are utilized for various tasks.
Obviously, each original domain data lacks certain kinds of fields, e.g., AG NEWS data only have two fields: category label and text.
Therefore, we leverage the T0 to predict the missing fields in order to conduct different kinds of tasks using the prompts having already trained in T0.
For some tasks we also train a specific model for prediction to get better performance.
After that, we filter samples according to the confidence score (i.e., the probability output by the model).
B M ORE EXPLORATIONS ON THE TRANSFER ABILITY B. S TATISTICS OF THE CURRECT DATASETS
We report the size of the dataset in our experiments in Table 7.
To explore more statistical features of the original dataset, we further provide some statistical indicators.
Specifically, we calculate the average sequence length (SLEN) and the mean segmental TTR (MSTTR) of the prompted input of training datasets.
MSTTR is calculated with a window size of 50.
We calculate those statistical values using at most 10000 examples for each prompted task for sake of time.
Results are in Table 7.
An interesting phenomenon is that datasets with fewer training examples are more likely to be a dataset with general transfer ability. We speculate that it is because a lot of manual effort is needed to make up those datasets with sufficient knowledge.
It is too costly to make these datasets extremely large.
It seems that the statistics such as MSTTR or data length alone cannot be good indicators for transfer ability.
More explorations about the influence factors of transfer ability are left for future work.
C M ORE R ESULTS ON O THER M ODELS AND O THER B ASELINES C. R ESULTS ON L ARGER M ODEL
To verify whether the observation that a small number of key tasks dominate zero-shot performance still holds true on larger models, we conduct the top-8 tasks experiments on T5-XL, similar to
Section 3.2.
Results are in Table 8.
From the results, we can see that the model trained on top-8 only slightly outperforms the baseline, while greatly defeating the performance of the model trained on all
T0 tasks without the top-8 tasks.
C. R ESULTS ON D ECODER -O NLY A RCHITECTURE
To verify whether the results still hold true on more architectures, we experiment on more architectures.
Considering that the encoder-only model is not suitable for language modeling tasks, here we only consider the decoder-only architecture.
In specific, we conduct the top-8 experiments on GPT-Neo1.3B (Black et al., 2021).
Results are in Table 9.
From the results, we can see that, the observation that a small number of key tasks dominate zero-shot performance mentioned in Section 3.2 still holds true. We train with a prefix-LM loss.
[Table]
MSTTR
In Len
Out Len
[Table]
[Table]
SamSum
[Table]
[Table]
[Table]
DBPedia
[Table]
[Table]
[Table]
C. R ESULTS ON O THER B ASELINES
Since most tasks in T0 are defined as QA tasks
[Table]
the observation that QA tasks are important might not be fair enough.
Thus, we want to investigate whether the statement that “some general transfer classes dominate the zero-shot performance” still holds with a totally different mixture of datasets.
Therefore, we consider conducting the experiment on the prompted datasets of FLAN (Wei et al., 2022).
Noted that reading comprehension is one of the task types used in FLAN, which can be regarded as narrative QA (in this way, we classify QA tasks based on the content rather than the format), so we hope to explore what will happen if we remove all reading comprehension tasks in the mixture of datasets used in FLAN.
We use exactly the same datasets and prompts as FLAN (Wei et al., 2022), except that we include three dialogue datasets in the same way as in FLAN-T5 (Chung et al., 2022), and exclude the translation datasets.
We leave the NLI and Commonsense Reasoning, as the hold-out test set, which follows
FLAN.
We conduct the experiments as follows: 1.
Training with all remaining tasks in FLAN.
(43 tasks in total); 2.
Training with only the reading comprehension tasks in FLAN.
(7 tasks in total); 3.
Training without the reading comprehension tasks in FLAN.
(36 tasks in total)
As can be seen in Table 10, the model which is trained on reading comprehension tasks greatly outperforms the model trained on FLAN tasks and FLAN tasks without reading comprehension tasks.
Therefore, these experiments serve as supplementary to our main experiment conducted on T datasets.
[Table]
D F ULL R ESULTS D. F ULL R ESULTS ON T OP -3 KEY DATASETS
The results when training on the top-3 key datasets for each test task are in Table 11 and Table 12.
We can see that the model trained on top-3 key datasets shows comparable results with the T0 baseline
. D. F ULL R ESULTS E VALUATED ON H ELD -O UT T EST D ATASETS
Full results evaluated on held-out test datasets are in Table 13 and Table 14.
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]