Original: F INETUNED L ANGUAGE M ODELS A RE Z ERO -S HOT L EARNERS
Revised: F INETUNED L ANGUAGE M ODELS A RE Z ERO -S HOT L EARNERS

Original: Anonymous authors Paper under double-blind review
Revised: Research

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper explores a simple method for improving the zero-shot learning abilities of language models.
Revised: This paper explores a simple method for improving the zero-shot learning abilities of language models.

Original: We show that instruction tuning —ﬁnetuning language models on a collection of datasets described via instructions—substantially boosts zeroshot performance on unseen tasks.
Revised: We show that instruction tuning —ﬁnetuning language models on a collection of datasets described via instructions—substantially improves zeroshot performance on unseen tasks.

Original: We take a 137B parameter pretrained language model and instruction tune it on over 60 NLP datasets verbalized via natural language instruction templates.
Revised: We take a 137B parameter pretrained language model and instruction tune it on over 60 NLP datasets verbalized via natural language instruction templates.

Original: We evaluate this instruction-tuned model, which we call FLAN, on unseen task types.
Revised: We evaluate this instruction-tuned model, which we call FLAN, on unseen task types.

Original: FLAN substantially improves the performance of its unmodiﬁed counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 datasets that we evaluate.
Revised: FLAN substantially improves the performance of its unmodiﬁed counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 datasets that we evaluate.

Original: FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC,
Revised: FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC,

Original: OpenbookQA and StoryCloze.
Revised: OpenbookQA, and StoryCloze.

Original: Ablation studies reveal that number of datasets and model scale are keycomponents to the success of instruction tuning
Revised: Ablation studies reveal that number of ﬁnetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.

Original: [Figure]
Revised: [Figure]

Original: Inference on unseen task type
Revised: Inference on unseen task type

Original: Finetune on many tasks (“instruction-tuning”)
Revised: Finetune on many tasks (“instruction- tuning”

Revised: [Figure]

Revised: Performance

Original: [Figure]
Revised: [Figure]

Revised: unseen task types

Original: [Table]
Revised: Natural language inference 42. Reading Comprehension 63. Closed-Book QA 49. Here

Original: [Figure]
Revised: [Figure]

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Language models (LMs) at scale, such as GPT-3 (Brown et al., 2020), have been shown to perform few-shot learning remarkably well.
Revised: Language models (LMs) at scale, such as GPT-3 (Brown et al., 2020), have been shown to perform few-shot learning remarkably well.

Original: They are less successful at zero-shot learning, however.
Revised: They are less successful at zero-shot learning, however.

Original: For example, GPT-3’s zero-shot performance is much worse than few-shot performance on tasks such as reading comprehension, question answering, and natural language inference.
Revised: For example, GPT-3’s zero-shot performance is much worse than few-shot performance on tasks such as reading comprehension, question answering, and natural language inference.

Original: One potential reason is that, without few-shot exemplars, it is harder for models to perform well on prompts that are not similar to the format of the pretraining data.
Revised: One potential reason is that, without few-shot exemplars, it is harder for models to perform well on prompts that are not similar to the format of the pretraining data.

Original: In this paper, we explore a simple method to improve the zero-shot performance of large language models, which would expand their reach to a broader audience.
Revised: In this paper, we explore a simple method to improve the zero-shot performance of large language models, which would expand their reach to a broader audience.

Original: We leverage the intuition that NLP tasks can be described via natural language instructions, such as “ Is the sentiment of this movie review positive or negative? ” or “ Translate ‘how are you’ into Chinese. ”
Revised: We leverage the intuition that NLP tasks can be described via natural language instructions, such as “ Is the sentiment of this movie review positive or negative? ” or “ Translate ‘how are you’ into Chinese. ”

Original: We take a pretrained language model of 137B parameters and perform instruction tuning —ﬁnetuning the model on a mixture of more than 60 NLP datasets expressed via natural language instructions.
Revised: We take a pretrained language model of 137B parameters and perform instruction tuning —ﬁnetuning the model on a mixture of more than 60 NLP datasets expressed via natural language instructions.

Original: We refer to this resulting model  FLAN, for Finetuned LAnguage Net.
Revised: We refer to this resulting model as FLAN, for Finetuned Language Net.

Original: To evaluate the zero-shot performance of FLAN on unseen tasks, we group NLP datasets into clusters based on their task types and hold out each cluster for evaluation while instruction tuning FLAN on all other clusters.
Revised: To evaluate the zero-shot performance of FLAN on unseen tasks, we group NLP datasets into clusters based on their task types and hold out each cluster for evaluation while instruction tuning FLAN on all other clusters.

Original: For example, as shown in Figure 1, to evaluate FLAN’s ability to perform natural language inference, we instruction tune the model on a range of other NLP tasks including commonsense reasoning, translation, and sentiment analysis.
Revised: For example, as shown in Figure 1, to evaluate FLAN’s ability to perform natural language inference, we instruction tune the model on a range of other NLP tasks such as commonsense reasoning, translation, and sentiment analysis.

Original: As this setup ensures that FLAN has not seen any natural language inference tasks in instruction tuning, we then evaluate its ability to perform zero-shot natural language inference.
Revised: As this setup ensures that FLAN has not seen any natural language inference tasks in instruction tuning, we then evaluate its ability to perform zero-shot natural language inference.

Original: Our evaluations show that FLAN substantially improves the zero-shot performance of the base 137B-parameter model.
Revised: Our evaluations show that FLAN substantially improves the zero-shot performance of the base 137B-parameter model.

Original: FLAN’s zero-shot also outperforms 175B-parameter GPT-3’s zero-shot onof 25 datasets that we evaluate, and even outperforms GPT-3’s few-shot by a large margin on ANLI,
Revised: FLAN’s zero-shot also outperforms 175B-parameter GPT-3’s zero-shot onof 25 datasets that we evaluate, and even outperforms GPT-3’s few-shot by a large margin on ANLI,

Original: RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze.
Revised: RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze.

Original: In ablation studies, we ﬁnd that increasing the number of task clusters in instruction tuning improves performance on unseen tasks and that the beneﬁts of instruction tuning emerge only with sufﬁcient model scale.
Revised: In ablation studies, we ﬁnd that increasing the number of task clusters in instruction tuning improves performance on unseen tasks and that the beneﬁts of instruction tuning emerge only with sufﬁcient model scale.

Original: Instruction tuning is a simple method that, as depicted in Figure 2, combines appealing aspects of both the pretrain–ﬁnetune and prompting paradigms by using supervision via ﬁnetuning to improve language model’s responses to inference-time text interactions.
Revised: Instruction tuning is a simple method that, as depicted in Figure 2, combines appealing aspects of both the pretrain–ﬁnetune and prompting paradigms by using supervision via ﬁnetuning to improve language model’s responses to inference-time text interactions.

Original: Our empirical results demonstrate promising abilities of language models to perform tasks described purely via instructions.
Revised: Our empirical results demonstrate promising abilities of language models to perform tasks described purely via instructions.

Revised: Source code for loading the instruction tuning dataset used for FLAN is publicly available at https://github.com/google-research/flan .

Original: [Figure]
Revised: [Figure]

Original:  FLAN: I NSTRUCTION T UNING I MPROVES Z ERO -S HOT L EARNING
Revised:  FLAN: I NSTRUCTION T UNING I MPROVES Z ERO -S HOT L EARNING

Original: The motivation of instruction tuning is to improve the ability of language models to respond to NLP instructions.
Revised: The motivation of instruction tuning is to improve the ability of language models to respond to NLP instructions.

Original: The idea is that by using supervision to teach an LM to perform tasks described via instructions, it will learn to follow instructions and do so even for unseen tasks.
Revised: The idea is that by using supervision to teach an LM to perform tasks described via instructions, the LM will learn to follow instructions and do so even for unseen tasks.

Original: To evaluatethe model’s performance on unseen tasks, we group datasets into clusters by task type and hold out each task cluster for evaluation while instruction tuning on all remaining clusters.
Revised: To evaluate performance on unseen tasks, we group datasets into clusters by task type and hold out each task cluster for evaluation while instruction tuning on all remaining clusters.

Original: 2. T ASKS & T EMPLATES
Revised: 2. T ASKS & T EMPLATES

Original: As creating an instruction tuning dataset with many tasks from scratch would be resource-intensive
Revised: As creating an instruction tuning dataset with many tasks from scratch would be resource-intensive

Original: [Table]
Revised: [Table]

Original: we transform existing datasets from the research community into an instructional format.
Revised: we transform existing datasets from the research community into an instructional format.

Original: We aggregate 62 text datasets that are publicly available on Tensorﬂow Datasets, including both language understanding and language generation tasks, into a single mixture.
Revised: We aggregate 62 text datasets that are publicly available on Tensorﬂow Datasets, including both language understanding and language generation tasks, into a single mixture.

Original: Figure 3 shows these datasets; each dataset is categorized into one of twelve task clusters
Revised: Figure 3 shows these datasets— each dataset is categorized into one of twelve task clusters

Original: [Table]
Revised: [Table]

Original: for which datasets in a given cluster are of the same task type.
Revised: for which datasets in a given cluster are of the same task type.

Original: Descriptions, sizes [Table] and examples of each dataset are shown in Appendix H
Revised: Descriptions, sizes, and examples of each dataset are shown in Appendix G

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: For each dataset, we manually compose ten unique templates that use natural language instructions to describe the task for that dataset.
Revised: For each dataset, we manually compose ten unique templates that use natural language instructions to describe the task for that dataset.

Original: While most of the ten templates describe the original task, to increase diversity, for each dataset we also include up to three templates that “turned the task around,” (e.g., for sentiment classiﬁcation we include templates asking to generate a movie review).
Revised: While most of the ten templates describe the original task, to increase diversity, for each dataset we also include up to three templates that “turned the task around,” (e.g., for sentiment classiﬁcation we include templates asking to generate a movie review).

Original: We then instruction tune a pretrained language model on the mixture of all datasets, with examples in each dataset formatted via a randomly selected instruction template for that dataset.
Revised: We then instruction tune a pretrained language model on the mixture of all datasets, with examples in each dataset formatted via a randomly selected instruction template for that dataset.

Original: Figure 4 shows multiple instruction templates for a natural language inference dataset.[Figure] hypothesis [Figure]
Revised: Figure 4 shows multiple instruction templates for a natural language inference dataset.

Original: <options

Original: [Figure]

Original: Template

Original: Template

Original: [Figure]

Original: hypothesis

Original: [Figure]
Revised: [Figure]

Original: from

Original: [Figure]

Original: premise:

Original: [Figure]

Original: : <premise

Original: [Figure]
Revised: [Figure]

Original: Hypothesis: <hypothesis

Original: [Figure]

Original: <options>

Original: Template

Original: Template 4,

Original: [Figure]
Revised: [Figure]

Original: 2. E VALUATION S PLITS
Revised: 2. E VALUATION S PLITS

Original: We are interested in how FLAN performs on tasks not seen in instruction tuning, and so it is crucial to deﬁne what counts as an unseen task.
Revised: We are interested in how FLAN performs on tasks not seen in instruction tuning, and so it is crucial to deﬁne what counts as an unseen task.

Original: Whereas some prior work deﬁnes unseen tasks by disallowing the same dataset to appear in training, we use a more conservative deﬁnition that leverages the task clusters from Figure 3.
Revised: Whereas some prior work deﬁnes unseen tasks by disallowing the same dataset to appear in training, we use a more conservative deﬁnition that leverages the task clusters from Figure 3.

Original: In this work, we only consider dataset D unseen at evaluation time if no datasets from any task clusters that D belongs to were seen during instruction tuning.
Revised: In this work, we only consider dataset D unseen at evaluation time if no datasets from any task clusters that D belongs to were seen during instruction tuning.

Original: For instance, if D is an entailment task, then no entailment datasets appeared in instruction tuning, and we instruction-tuned on all other clusters.
Revised: For instance, if D is an entailment task, then no entailment datasets appeared in instruction tuning, and we instruction-tuned on all other clusters.

Original: 1 Hence, to evaluate zero-shot FLAN on c task clusters, we instruction tune c models, where each model holds out a different task cluster for evaluation.
Revised: 1 Hence, to evaluate zero-shot FLAN on c task clusters, we instruction tune c models, where each model holds out a different task cluster for evaluation.

Original: 2. C LASSIFICATION WITH O PTIONS
Revised: 2. C LASSIFICATION WITH O PTIONS

Original: The output space for a given task is either one of several classes (classiﬁcation) or free text (generation).
Revised: The output space for a given task is either one of several classes (classiﬁcation) or free text (generation).

Original: As FLAN is an instruction-tuned version of a decoder-only language model, it naturally responds in free text, and so no further modiﬁcations are needed for generation tasks.
Revised: As FLAN is an instruction-tuned version of a decoder-only language model, it naturally responds in free text, and so no further modiﬁcations are needed for generation tasks.

Original: For classiﬁcation tasks, prior work (Brown et al., 2020)has used a rank classiﬁcation approach where, for example, only two outputs (“ yes ” and “ no ”) are considered and the higher probability one is taken as the model’s prediction.
Revised: For classiﬁcation tasks, prior work (Brown et al., 2020) used a rank classiﬁcation approach where, for example, only two outputs (“ yes ” and “ no ”) are considered and the higher probability one is taken as the model’s prediction.

Original: Though this procedure is logically sound, it is imperfect in that the probability mass for answers may have an undesired distribution among ways of saying each answer (e.g., a large number of alternative ways of saying “ yes ” may lower the probability mass assigned to “ yes ”).
Revised: Though this procedure is logically sound, it is imperfect in that the probability mass for answers may have an undesired distribution among ways of saying each answer (e.g., a large number of alternative ways of saying “ yes ” may lower the probability mass assigned to “ yes ”).

Original: Therefore, we include an options sufﬁx, in which we append the token OPTIONS to the end of a classiﬁcation task along with a list of the output classes for that task.
Revised: Therefore, we include an options sufﬁx, in which we append the token OPTIONS to the end of a classiﬁcation task along with a list of the output classes for that task.

Original: This makes the model aware of which choices are desired when responding to classiﬁcation tasks.
Revised: This makes the model aware of which choices are desired when responding to classiﬁcation tasks.

Original: Example use of options is shown in the NLI and commonsense examples in Figure 1.
Revised: Example use of options is shown in the NLI and commonsense examples in Figure 1.

Original: 2. T RAINING D ETAILS
Revised: 2. T RAINING D ETAILS

Original: Model architecture and pretraining.
Revised: Model architecture and pretraining.

Original: In our experiments, we use  a dense left-to-right, decoder-only transformer language model of 137B parameters.
Revised: In our experiments, we use LaMDA-PT, a dense left-to-right, decoder-only transformer language model of 137B parameters (Thoppilan et al., 2022).

Original: This model is pretrained on a collection of web documents (including those with computer code), dialog data, and Wikipedia, tokenized into 2.49T BPE tokens with a 32k vocabulary using the SentencePiece library (Kudo & Richardson, 2018).
Revised: This model is pretrained on a collection of web documents (including those with computer code), dialog data, and Wikipedia, tokenized into 2.49T BPE tokens with a 32k vocabulary using the SentencePiece library (Kudo & Richardson, 2018).

Original: Around 10% of the pretraining data was non-English.
Revised: Around 10% of the pretraining data was non-English.

Original: We refer to this pretrained model as Base LM .
Revised: Note that LaMDA-PT only has language model pretraining (c.f. LaMDA, which was ﬁnetuned for dialog).

Original: Instruction tuning procedure.
Revised: Instruction tuning procedure.

Original: FLAN is the instruction-tuned version of Base LM.
Revised: FLAN is the instruction-tuned version of LaMDA-PT.

Original: Our instruction tuning pipeline mixes all datasets and randomly samples from each dataset.
Revised: Our instruction tuning pipeline mixes all datasets and randomly samples from each dataset.

Original: To balance the different sizes of datasets, we limit the number of training examples per dataset to 30k and follow the examplesproportional mixing scheme (Raffel et al., 2020) with a mixing rate maximum of 3k.
Revised: To balance the different sizes of datasets, we limit the number of training examples per dataset to 30k and follow the examples-proportional mixing scheme (Raffel et al., 2020) with a mixing rate maximum of 3k.

Original: We ﬁnetune all models for 30k gradient steps with a batch size of 8,192 tokens using the Adafactor Optimizer (Shazeer & Stern, 2018) with a learning rate of 3e-5.
Revised: We ﬁnetune all models for 30k gradient steps with a batch size of 8,192 tokens using the Adafactor Optimizer (Shazeer & Stern, 2018) with a learning rate of 3e-5.

Original: The input and target sequence lengths used in ﬁnetuning are 1024 and 256, respectively.
Revised: The input and target sequence lengths used in ﬁnetuning are 1024 and 256, respectively.

Original: We use packing (Raffel et al., 2020) to combine multiple training examples into a single sequence, separating inputs from targets using a special EOS token.
Revised: We use packing (Raffel et al., 2020) to combine multiple training examples into a single sequence, separating inputs from targets using a special EOS token.

Original: This instruction tuning takes around 60 hours on a TPUv3 with 128 cores.
Revised: This instruction tuning takes around 60 hours on a TPUv3 with 128 cores.

Original: For all evaluations, we report results on the ﬁnal checkpoint trained for 30k steps.
Revised: For all evaluations, we report results on the ﬁnal checkpoint trained for 30k steps.

Original:  R ESULTS
Revised:  R ESULTS

Original: We evaluate FLAN on natural language inference, reading comprehension, closed-book QA, translation, commonsense reasoning, coreference resolution, and struct-to-text.
Revised: We evaluate FLAN on natural language inference, reading comprehension, closed-book QA, translation, commonsense reasoning, coreference resolution, and struct-to-text.

Original: As described in §2.2, we evaluate on unseen tasks by grouping datasets into task clusters and holding out each cluster for evaluation while instruction tuning on all remaining clusters (i.e., each evaluation task cluster uses a different checkpoint).
Revised: As described in §2.2, we evaluate on unseen tasks by grouping datasets into task clusters and holding out each cluster for evaluation while instruction tuning on all remaining clusters (i.e., each evaluation task cluster uses a different checkpoint).

Original: For each dataset, we report the meanand standard deviation of performance on all templates, which proxies the expected performance given a typical natural language instruction.
Revised: For each dataset, we evaluate the mean of performance on all templates, which proxies the expected performance given a typical natural language instruction.

Original: As a dev set is sometimes available for manual prompt engineering (Brown et al., 2020), for each dataset we also report the test set performance using the template with the best dev set performance.
Revised: As a dev set is sometimes available for manual prompt engineering (Brown et al., 2020), for each dataset we also obtain the test set performance using the template with the best dev set performance.

Original: For comparison, we show the zero-shot and few-shot performance of GPT-3 175B, asreported in their paper.
Revised: For comparison, we report zero and few-shot results for LaMDA-PT using the same prompts as

Original: With the best dev template, zero-shot FLAN outperforms zero-shot GPT-3 on 20 out of 25 tasks and even surpasses GPT-3’s few-shot performance on 10 tasks.

Original: We also report zero and few-shot results from Base LM using the same prompts as GPT-3 (as Base LM is not suitable for natural instructions without instruction tuning).
Revised: GPT-3 (as LaMDA-PT is not suitable for natural instructions without instruction tuning).

Original: This baseline provides the most direct ablation of how much instruction tuning helps.
Revised: This baseline provides the most direct ablation of how much instruction tuning helps.

Original: Instruction tuning signiﬁcantly improves Base LM on most tasks.
Revised: Instruction tuning signiﬁcantly improves LaMDA-PT on most datasets.

Original: In terms of supervised models, we report the performance of T5-11B (Raffel et al., 2020) when available; otherwise, we show the performance of BERT-large (Devlin et al., 2019).
Revised: We also show the zero-shot performances of GPT-3 175B (Brown et al., 2020) and GLaM 64B/64E (Du et al., 2021), as reported in their respective papers. With the best dev template, zero-shot outperforms zero and even surpasses -3’s few-shot performance on zero outperforms zero

Revised: GLaM

Original: Overall, we observe that instruction tuning is very effective on tasks naturally verbalized as instructions (e.g., NLI, QA, translation, struct-to-text) and is less effective on tasks directly formulated as language modeling, where instructions would be largely redundant (e.g., commonsense reasoning and coreference resolution tasks that are formatted as ﬁnishing an incomplete sentence or paragraph). 
Revised: Overall, we observe that instruction tuning is very effective on tasks naturally verbalized as instructions (e.g., NLI, QA, translation, struct-to-text) and is less effective on tasks directly formulated as language modeling, where instructions would be largely redundant (e.g., commonsense reasoning and coreference resolution tasks that are formatted as ﬁnishing an incomplete sentence or paragraph). Results on natural language inference, reading comprehension, closed-book QA, and translation are summarized in Figure 5 and described below. [Figure] OBQA

Revised: [Figure]

Original: 3. N ATURAL L ANGUAGE I NFERENCE
Revised: MultiRC BoolQ Translation EN to FR EN to DE

Revised: FR to EN

Revised: DE to EN

Revised: EN to RO

Revised: RO to EN

Revised: LaMDA-PT137B

Original: Table 1 shows results for natural language inference (NLI), where a model must determine whether a hypothesis is true given some premise.
Revised: FLAN 137B GPT [Figure] 175B

Revised: GLaM 64B

Original:  FLAN has strong performance in all cases.
Revised: [Figure] 64E [Figure] Performance of FLAN is the mean of up to 10 instructional templates per task.

Original: Despite high variance among different templates for CB (De Marneffe et al., 2019) and RTE (Wang et al., 2018),
Revised: Supervised models were either T5, BERT, or translation models (speciﬁed in Table 2 and Table 1 in the Appendix).

Original:  FLAN outperforms zero and few-shot GPT-3 for four datasets by a large margin without any prompt engineering. With the best dev template, FLAN outperforms few-shot GPT-3 on all ﬁve datasets.
Revised: Natural language inference (NLI). On ﬁve NLI datasets, where a model must determine whether a hypothesis is true given some premise, FLAN outperforms all baselines by a large margin.

Original: As noted by Brown et al.
Revised: As noted by Brown et al.

Original: (2020), perhaps one reason why GPT-3 struggles with NLI is that NLI examples are unlikely to have appeared naturally in an unsupervised training set and are thus awkwardly phrased as a continuation of a sentence.
Revised: (2020), perhaps one reason why GPT-3 struggles with NLI is that NLI examples are unlikely to have appeared naturally in an unsupervised training set and are thus awkwardly phrased as a continuation of a sentence.

Original: For FLAN [Table] we phrase NLI as the more natural question “ Does <premise> mean that <hypothesis>? ” and achieve much higher performance.[Table]
Revised: For FLAN, we phrase NLI as the more natural question “ Does <premise> mean that <hypothesis>? ”, achieving much higher performance.

Original: Results for reading comprehension, where models are asked to answer a question about a provided passage, are shown in Table 2.
Revised: Reading comprehension. On reading comprehension, where models are asked to answer a question about a provided passage, FLAN outperforms baselines for MultiRC (Khashabi et al., 2018) and

Original:  FLAN outperforms GPT-3 by a large margin on BoolQ (Clark et al., 2019a) and OBQA (Mihaylov et al., 2018), although Base LM already achieves high performance on BoolQ.On MultiRC (Khashabi et al., 2018), the performance of FLAN when using the best dev template is slightly higher than few-shot GPT-3. [Table]
Revised: OBQA (Mihaylov et al., 2018). On BoolQ (Clark et al., 2019a), FLAN outperforms GPT-3 by a large margin, though LaMDA-PT already achieves high performance on BoolQ.

Original: Table 2 also shows results for closed-book QA, which asks models to answer questions about the world without access to speciﬁc information containing the answer.
Revised: Closed-book QA. For closed-book QA, which asks models to answer questions about the world without access to speciﬁc information containing the answer, FLAN outperforms GPT-3 on all four datasets.

Original: On ARC-easy and ARC-challenge (Clark et al., 2018), FLAN substantially outperforms both zero-shot andfew-shot GPT-3.
Revised: Compared to GLaM, FLAN has better performance on ARC-e and

Original: For Natural
Revised: ARC-c

Original: Questions (NQ; Lee et al., 2019; Kwiatkowski et al., 2019) and TQA (Joshi et al., 2017), FLAN outperforms zero-shot GPT-3 but does not outperform few-shot GPT-3.
Revised: (Clark et al., 2018), and slightly lower performance on NQ (Lee et al., 2019; Kwiatkowski et al., 2019) and TQA (Joshi et al., 2017).

Original: 3. T RANSLATION
Revised: Translation.

Original: Similar to GPT-3, the training data for Base LM is around 90% English and includes some text in other languages that was not speciﬁcally used to train the model to perform machine translation.
Revised: Similar to GPT-3, the training data for LaMDA-PT is around 90% English and includes some text in other languages that was not speciﬁcally used to train the model to perform machine translation.

Original: We also evaluate FLAN’s performance on machine translation for the three datasets evaluated in the GPT-3 paper: French–English from WMT’14 (Bojar et al., 2014), and German–English and
Revised: We also evaluate FLAN’s performance on machine translation for the three datasets evaluated in the GPT-3 paper: French–English from WMT’14 (Bojar et al., 2014), and German–

Original:  Romanian–English from WMT’16 (Bojar et al., 2016).
Revised: English and Romanian–English from WMT’16 (Bojar et al., 2016).

Original: These results are shown in Table 3.

Original: For Base LM, zero-shot translation had low performance, but few-shot translation results were comparable with GPT-3; FLAN outperforms few-shot Base LM in ﬁve of the six evaluations.

Original: Compared with GPT-3, FLAN outperforms zero-shot GPT-3 for all six evaluations, though it underperforms few-shot GPT-3 in most cases.
Revised: Compared with GPT-3, FLAN outperforms zero-shot GPT-3 for all six evaluations, though it underperforms few-shot GPT-3 in most cases.

Original: Similar to GPT-3, FLAN shows strong results for translating into English and compares favorably against supervised translation baselines.
Revised: Similar to GPT-3, FLAN shows strong results for translating into English and compares favorably against supervised translation baselines.

Original: Translating from English into other languages, however, was relatively weaker, as might be expected given that FLAN uses an English sentencepiece tokenizer and that the majority of pretraining data is English.[Table] En → Fr BLEU Fr → En BLEU
Revised: Translating from English into other languages, however, was relatively weaker, as might be expected given that FLAN uses an English sentencepiece tokenizer and that the majority of pretraining data is English.

Original: En → De BLEU De → En BLEU En → Ro BLEU Ro → En BLEU
Revised: Additional tasks.

Original: [Table] 3. A DDITIONAL T ASKS Although we see strong results for the above task clusters, one limitation with instruction tuning is that it does not improve performance for many language modeling tasks (e.g., commonsense reasoning or coreference resolution tasksthat are formulated as sentence completions).
Revised: Although we see strong results for the above task clusters, one limitation with instruction tuning is that it does not improve performance for many language modeling tasks (e.g., commonsense reasoning or coreference resolution tasks formulated as sentence completions).

Original: To summarize the full result shown in Table 4 in Appendix A, for seven commonsense reasoning and coreference resolutiontasks, FLAN only outperforms Base LM on three of the seven tasks (although performance was comparable on most tasks). With the best dev template, FLAN only outperforms zero-shot GPTon two of the seven tasks.
Revised: For seven commonsense reasoning and coreference resolution tasks (see Table 2 in the Appendix), FLAN only outperforms LaMDA-PT on three of the seven tasks.

Original: This negative result indicates that when the downstream task is the same as the original language modeling pre-training objective (i.e., in cases where instructions are largely redundant), instruction tuning is not useful.
Revised: This negative result indicates that when the downstream task is the same as the original language modeling pre-training objective (i.e., in cases where instructions are largely redundant), instruction tuning is not useful.

Original: Finally, we report results forthe sentiment analysis, paraphrase detection, and struct to text task clusters, as well as other datasets for which GPT-3 results are not available, in Table 5 and Table 6 in 
Revised: Finally, we report results for sentiment analysis, paraphrase detection, and struct-to-text, as well as additional datasets for which GPT-3 results are not available, in Table 2 and Table 1 in the Appendix.

Original: Appendix A. Generally,we see that zero-shot FLANsubstantially outperforms zero-shot Base LM and is comparable with or better than few-shot Base LM, though FLAN’s performance is still far from that of supervised models on most tasks.
Revised: Generally, zero-shot FLAN outperforms zero-shot LaMDA-PT and is comparable with or better than few-shot LaMDA-PT.

Original:  A BLATION S TUDIES & F URTHER A NALYSIS 4. N UMBER OF INSTRUCTION TUNING CLUSTERS
Revised:  A BLATION S TUDIES & F URTHER A NALYSIS 4. N UMBER OF INSTRUCTION TUNING CLUSTERS

Original: As the core question of our paper asks how instruction tuning improves a model’s zero-shot performance on unseen tasks, in this ﬁrst ablation we examine how performance is affected by the number of clusters and tasks used in instruction tuning.
Revised: As the core question of our paper asks how instruction tuning improves a model’s zero-shot performance on unseen tasks, in this ﬁrst ablation we examine how performance is affected by the number of clusters and tasks used in instruction tuning.

Original: For this setup, we hold out NLI, closed-book QA, and commonsense reasoning as evaluation clusters, and use the seven remaining clusters for instruction tuning.
Revised: For this setup, we hold out NLI, closed-book QA, and commonsense reasoning as evaluation clusters, and use the seven remaining clusters for instruction tuning.

Original: 2 We show results for 1-7 instruction tuning clusters, where clusters are added in decreasing order of number of tasks per cluster.
Revised: 3 We show results for one to seven instruction tuning clusters, where clusters are added in decreasing order of number of tasks per cluster

Revised: [Figure]

Original: Figure 5 shows these results.
Revised: Figure 6 shows these results.

Original: As expected, we observe that average performance across the three held-out clusters improves as we add additional clusters and tasks to instruction tuning (with the exception of the sentiment analysis cluster), conﬁrming the beneﬁts of our proposed instruction tuning approach on zero-shot performance on novel tasks.
Revised: As expected, we observe that average performance across the three held-out clusters improves as we add additional clusters and tasks to instruction tuning (with the exception of the sentiment analysis cluster), conﬁrming the beneﬁts of our proposed instruction tuning approach on zero-shot performance on novel tasks.

Original: It is further interesting to see that, for the seven clusters we test, the performance does not appear to saturate, implying that performance may further improve with even more clusters added to instruction tuning.
Revised: It is further interesting to see that, for the seven clusters we test, the performance does not appear to saturate, implying that performance may further improve with even more clusters added to instruction tuning.

Original: Of note, this ablation does not allow us to draw conclusions about which instruction tuning cluster contributes the most to each evaluation cluster, although we see minimal added value from the sentiment analysis cluster.
Revised: Of note, this ablation does not allow us to draw conclusions about which instruction tuning cluster contributes the most to each evaluation cluster, although we see minimal added value from the sentiment analysis cluster.

Original: [Equation]
Revised: [Figure]

Original: Clusters used for instruction tuning

Original:  [Figure][Table]
Revised: [Equation] [Figure]

Original: 30 # clusters

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: o
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: e n c e

Original: [Equation]
Revised: [Figure]

Original: Q

Original: [Figure]
Revised: [Figure]

Original: 4. S CALING LAWS
Revised: 4. S CALING LAWS

Revised: [Figure]

Revised: it actually hurts generalization to unseen tasks, potentially because all model capacity is used to learn the mixture of instruction tuning tasks

Original: As Brown et al.
Revised: As Brown et al.

Original: (2020) shows that zero and few-shot capabilities of language models substantially improve for larger models, we next explore how the beneﬁts of instruction tuning are affected by model scale.
Revised: (2020) shows that zero and few-shot capabilities of language models substantially improve for larger models, we next explore how the beneﬁts of instruction tuning are affected by model scale.

Original: Using the same cluster split as in the previous ablation study, we evaluate the effect of instruction tuning on models of size 422M, 2B, 8B, 68B, and 137B parameters.
Revised: Using the same cluster split as in the previous ablation study, we evaluate the effect of instruction tuning on models of size 422M, 2B, 8B, 68B, and 137B parameters.

Original: Figure 6 shows these results.
Revised: Figure 7 shows these results.

Original: First, as a check of correctness, we conﬁrm in Figure 6A that, for all model sizes, performance on tasks that were seen during instruction tuning improves by a large margin over the untuned model without instruction tuning, as would be highly expected.

Original: As for held-out tasks, in Figure 6B we see that for the two models on the order of 100B parameters, instruction tuning substantially improves performance, as isalso expected given the prior results in our paper.
Revised: We see that for the two models on the order of 100B parameters, instruction tuning substantially improves performance on held-out tasks, as is expected given the prior results in our paper.

Original: The behavior on held-out tasks for the 8B and smaller models, however, is thought-provoking—instruction tuning actually hurts performance on held-out tasks.
Revised: The behavior on held-out tasks for the 8B and smaller models, however, is thoughtprovoking—instruction tuning actually hurts performance on held-out tasks.

Original: One potential explanation for this result could be that for small-scale models, learning the ∼ 40 tasks used during instruction tuning ﬁlls the entire model capacity, causing these models to perform worse on new tasks.
Revised: One potential explanation for this result could be that for small-scale models, learning the ∼ 40 tasks used during instruction tuning ﬁlls the entire model capacity, causing these models to perform worse on new tasks.

Original: For the larger scale models, instruction tuning ﬁlls up some model capacity but also teaches these models the ability to follow instructions, allowing them to generalize to new tasks with the remaining capacity. 
Revised: Under this potential explanation, for the larger scale models, instruction tuning ﬁlls up some model capacity but also teaches these models how to follow instructions, allowing them to generalize to new tasks with the remaining capacity. 4. R OLE OF INSTRUCTIONS

Original: [Figure]
Revised: [Figure]

Original: 2B
Revised: In a ﬁnal ablation study, we explore the role of instructions during ﬁnetuning, as one possibility is that performance gains come entirely from multi-task ﬁnetuning and the model could perform just as well without instructions.

Original: [Figure] [Figure] Untuned [Figure]
Revised: We hence consider two ﬁnetuning setups without instructions.

Original: B
Revised: In a no template setup, only inputs and outputs were given to the model (e.g., for translation the input would be “ The dog runs. ” and the output would be “ Le chien court. ”).

Original: A v e r age z e r o - s ho t a cc u r a cy onhe l d - ou t t a sks ( % ) Performance on held-out tasks 0.4B 2B 8B
Revised: In a dataset name setup, each input is prepended with the name of the task and dataset (e.g., for translation to French, the input would be “ [Translation: WMT’14 to French]

Original: [Figure]
Revised: The dog runs. ”).

Original: 137B Model Size (# parameters) A v e r age a cc u r a cy onhe l d - i n t a sks ( % ) ( R ead i
Revised: We compare these two ablations to FLAN’s ﬁnetuning procedure, which used natural instructions (e.g., “ Please translate this sentence to French: ‘The dog runs.’ ”).

Original: ng c o m p r ehen s i on , s en t i m [Figure]
Revised: We perform evaluations for four held-out clusters from Figure 5.

Original: t
Revised: For the no template setup, we used the FLAN instructions during zero-shot inference (because if we used no template, the model would not know what task to perform).

Original: [Figure] The effect of instruction tuning on  performance on unseen tasks depends on the model scale. Whereas instruction tuning helps large models generalize to new tasks, for small models it actually hurts generalization to unseentasks, potentially because all model capacity is used to learn the mixture of instruction tuning tasks.
Revised: For models ﬁnetuned on dataset name only, we report zero-shot performance for FLAN instructions as well as using the dataset name. Figure 8 shows the results—both ablation conﬁgurations performed substantially worse than FLAN, indicating that training with instructions is crucial for zero-shot performance on unseen tasks.

Original: 4. I NSTRUCTIONS WITH F EW -S HOT E XEMPLARS
Revised: 4. I NSTRUCTIONS WITH F EW -S HOT E XEMPLARS

Original: So far, we have focused on instruction tuning in the zero-shot setting.
Revised: So far, we have focused on instruction tuning in the zero-shot setting.

Original: Here, we study how instruction tuning can be used when few-shot exemplars are available at inference time.
Revised: Here, we study how instruction tuning can be used when few-shot exemplars are available at inference time.

Original: The format for the few-shot setting builds on the zero-shot format.
Revised: The format for the few-shot setting builds on the zero-shot format.

Original: For some input x and output y , let instruct ( x ) denote the zero-shot instructions.
Revised: For some input x and output y , let instruct ( x ) denote the zero-shot instructions.

Original: Then, given k few-shot exemplars ( x i , y i ) ki =1 and a new input x , the instruction format for the few-shot setting is “instruct ( x 1 ) ⊕ y 1 ⊕ instruct ( x 2 ) ⊕ y 2 ⊕ . . .
Revised: Then, given k few-shot exemplars ( x i , y i ) ki =1 and a new input x , the instruction format for the few-shot setting is “instruct ( x 1 ) ⊕ y 1 ⊕ instruct ( x 2 ) ⊕ y 2 ⊕ . . .

Original: ⊕ instruct ( x k ) ⊕ y k ⊕ instruct ( x ) ”, where ⊕ denotes string concatenation with a delimiter token inserted in between.
Revised: ⊕ instruct ( x k ) ⊕ y k ⊕ instruct ( x ) ”, where ⊕ denotes string concatenation with a delimiter token inserted in between.

Original: At both training and inference time, exemplars are randomly drawn from the training set, and the number of exemplars is capped at 16 and such that the total sequence length is less thantokens.
Revised: At both training and inference time, exemplars are randomly drawn from the training set, and the number of exemplars is capped at 16 and such that the total sequence length is less thantokens.

Original: Our experiment uses the same task splits and evaluation procedure as §3, such that few-shot exemplars for an unseen task are only used at inference time.
Revised: Our experiment uses the same task splits and evaluation procedure as §3, such that few-shot exemplars for an unseen task are only used at inference time.

Original: As shown in Figure 7, few-shot exemplars improve the performance on all task clusters, compared with zero-shot FLAN.
Revised: As shown in Figure [Figure] few-shot exemplars improve the performance on all task clusters, compared with zero-shot FLAN.

Original: Exemplars are especially effective for tasks with large/complex output spaces, such as struct to text, translation, and closed-book QA, potentially because exemplars help the model better understand the output format.
Revised: Exemplars are especially effective for tasks with large/complex output spaces, such as struct to text, translation [Figure] and closed-book QA [Figure]

Revised: potentially because exemplars help the model better understand the output format.

Original: In addition, for all task clusters, standard deviation among templates is lower for few-shot FLAN, indicating reduced sensitivity to prompt engineering.
Revised: In addition, for all task clusters, standard deviation among templates is lower for few-shot FLAN, indicating reduced sensitivity to prompt engineering

Original: [Figure]
Revised: [Figure]

Revised: . 4. I NSTRUCTION T UNING F ACILITATES P ROMPT T UNING

Revised: [Figure]

Revised: When prompt tuning on a given dataset, no tasks from the same cluster as that dataset were seen during instruction tuning.

Revised: Performance shown is the average on the SuperGLUE dev set.

Revised: As we’ve seen that instruction tuning improves the ability of a model to respond to instructions, it follows that, if FLAN is indeed more amenable to performing NLP tasks, then it should also achieve better performance when performing inference using soft prompts, represented by prepended continuous variables optimized via prompt tuning (Li & Liang, 2021; Lester et al., 2021).

Revised: As further analysis, we train continuous prompts for each of the SuperGLUE (Wang et al., 2019a) tasks in accordance with the cluster splits from §2.2 such that when prompt-tuning on task

Revised: T , no tasks in the same cluster as T were seen during instruction tuning.

Revised: Our prompt tuning setup follows the procedure of Lester et al.

Revised: (2021) except that we use a prompt length of 10, weight decay of 1e-4, and did not use dropout on the attention scores; we found in preliminary experiments that these changes improved the performance of LaMDA-PT.

Revised: Figure 10 shows the results of these prompt tuning experiments for both using a fully-supervised training set and in a low-resource setting with only 32 training examples.

Revised: We see that in all scenarios, prompt tuning works better with FLAN than LaMDA-PT.

Revised: In many cases, especially for the low-resource setting, prompt tuning on FLAN even achieves more than 10% improvement over prompt tuning on the LaMDA-PT.

Revised: This result exempliﬁes in another way how instruction tuning can result in a checkpoint that is more desirable for performing NLP tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work relates to several broad research areas including zero-shot learning, prompting, multi-task learning, and language models for NLP applications (Radford et al., 2019; Raffel et al., 2020;
Revised: Our work relates to several broad research areas including zero-shot learning, prompting, multi-task learning, and language models for NLP applications (Radford et al., 2019; Raffel et al., 2020;

Original: Brown et al., 2020; Efrat & Levy, 2020; Aghajanyan et al., 2021; Li & Liang, 2021, inter alia ).
Revised: Brown et al., 2020; Efrat & Levy, 2020; Aghajanyan et al., 2021; Li & Liang, 2021, inter alia ).

Original: We describe prior work for these broad areas in an extended related work section (Appendix E), and here we describe two subareas narrower in scope that perhaps relate most closely to our work.
Revised: We describe prior work for these broad areas in an extended related work section (Appendix D), and here we describe two subareas narrower in scope that perhaps relate most closely to our work.

Original: The way we ask a model to respond to instructions is similar to QA-based task formulation (Kumar et al., 2016; McCann et al., 2018), which aims to unify NLP tasks by casting them as QA over a context.
Revised: The way we ask a model to respond to instructions is similar to QA-based task formulation (Kumar et al., 2016; McCann et al., 2018), which aims to unify NLP tasks by casting them as QA over a context.

Original: Though these methods are very similar to ours, they mostly focus on multi-task learning instead of zero-shot learning, and—as noted by Liu et al.
Revised: Though these methods are very similar to ours, they mostly focus on multi-task learning instead of zero-shot learning, and—as noted by Liu et al.

Original: (2021)—they are generally not motivated by using existing knowledge in pretrained LMs.
Revised: (2021)—they are generally not motivated by using existing knowledge in pretrained LMs.

Original: Moreover, our work supercedes recent work such as
Revised: Moreover, our work supercedes recent work such as

Original: Chai et al.
Revised: Chai et al.

Original: (2020) and Zhong et al.
Revised: (2020) and Zhong et al.

Original: (2021) in terms of both model scale and scope of tasks.
Revised: (2021) in terms of both model scale and scope of tasks.

Original: The success of language models has led to nascent research on the ability of models to follow instructions.
Revised: The success of language models has led to nascent research on the ability of models to follow instructions.

Original: Most recently, Mishra et al.
Revised: Most recently, Mishra et al.

Original: ﬁnetune 140M parameter BART on instructions with few-shot exemplars, and evaluate its few-shot abilities on unseen tasks—this isvery similar to our few-shot instruction tuning result from §4.3.
Revised: ﬁnetune 140M parameter BART on instructions with few-shot exemplars, and evaluate its few-shot abilities on unseen tasks —this is similar to our few-shot instruction tuning result from §4.4.

Original: This promising result (as well as one from Ye et al.
Revised: This promising result (as well as one from Ye et al.

Original: (2021), which does not emphasize instructions as much) suggests that ﬁnetuning on a collection of tasks improves few-shot performance on unseen tasks, even at a smaller model scale.
Revised: (2021), which does not emphasize instructions as much) suggests that ﬁnetuning on a collection of tasks improves few-shot performance on unseen tasks, even at a smaller model scale.

Revised: Sanh et al.

Revised: (2021) ﬁnetune T5 in a setup similar to ours, ﬁnding that zero-shot learning can be improved in a model of 11B parameters.

Original: At a model scale similar to ours, OpenAI has a production version of GPT-3 called the “Instruct Series”, which, although they have not yet released a research paper, appears to have goals similar to our work.
Revised: At a model scale similar to ours, OpenAI’s InstructGPT models are trained via both ﬁnetuning and reinforcement learning to produce outputs that are more preferred by human raters (Ouyang et al., 2022).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Our paper has explored a simple question in zero-shot prompting: does ﬁnetuning a model on a collection of tasks phrased as instructions improve its performance on unseen tasks?
Revised: Our paper has explored a simple question in zero-shot prompting: does ﬁnetuning a model on a collection of tasks phrased as instructions improve its performance on unseen tasks?

Original: We operationalize this question via instruction tuning, a simple method that combines appealing aspects of both the pretrain–ﬁnetune and prompting paradigms.
Revised: We operationalize this question via instruction tuning, a simple method that combines appealing aspects of both the pretrain–ﬁnetune and prompting paradigms.

Original: Our instruction-tuned model, FLAN, improves performance against an untuned model and surpasses zero-shot GPT-3 on the majority of tasks that we evaluate on.
Revised: Our instruction-tuned model, FLAN, improves performance against an untuned model and surpasses zero-shot GPT-3 on the majority of tasks that we evaluate on.

Original: Ablation studies reveal that performance on unseen tasks improves with the number of instruction tuning task clusters, and, interestingly, that performance improvements from instruction tuning emerge only with sufﬁcient model scale.
Revised: Ablation studies reveal that performance on unseen tasks improves with the number of instruction tuning task clusters, and, interestingly, that performance improvements from instruction tuning emerge only with sufﬁcient model scale.

Original: Moreover, instruction tuning can be combined with other prompting methods such as few-shot prompting (as well as prompt tuning; see Appendix C).
Revised: Moreover, instruction tuning can be combined with other prompting methods such as few-shot prompting and prompt tuning.

Original: The diverse capabilities of language models at scale have drawn attention to the tradeoffs between specialist models (one model per task) and generalist models (one model for many tasks; Arivazhagan et al., 2019; Pratap et al., 2020), for which our study has potential implications.
Revised: The diverse capabilities of language models at scale have drawn attention to the tradeoffs between specialist models (one model per task) and generalist models (one model for many tasks; Arivazhagan et al., 2019; Pratap et al., 2020), for which our study has potential implications.

Original: Although one might expect labeled data to have the most natural role in improving specialist models, instruction tuning demonstrates how labeled data can be used to help large language models perform many, unseen tasks.
Revised: Although one might expect labeled data to have the most natural role in improving specialist models, instruction tuning demonstrates how labeled data can be used to help large language models perform many, unseen tasks.

Original: In other words, the positive effect of instruction tuning on cross-task generalization shows that task-speciﬁc training is complementary to general language modeling and motivates further research on generalist models.
Revised: In other words, the positive effect of instruction tuning on cross-task generalization shows that task-speciﬁc training is complementary to general language modeling and motivates further research on generalist models.

Original: As for limitations of our study, there is a degree of subjectivity in assigning tasks to clusters (though we try to use accepted categorizations in the literature), and we only explore the use of relatively short instructions of typically a single sentence (c.f. detailed instructions given to crowd-workers).
Revised: As for limitations of our study, there is a degree of subjectivity in assigning tasks to clusters (though we try to use accepted categorizations in the literature), and we only explore the use of relatively short instructions of typically a single sentence (c.f. detailed instructions given to crowd-workers).

Original: A limitation for our evaluation is that individual examples might have appeared in the models’ pretraining data, which includes web documents, though in post-hoc analysis (Appendix D) we do not ﬁnd
Revised: A limitation for our evaluation is that individual examples might have appeared in the models’ pretraining data, which includes web documents, though in post-hoc analysis (Appendix C) we do not ﬁnd

Original: any evidence that data overlap substantially impacted the results.
Revised: any evidence that data overlap substantially impacted the results.

Original: Finally, the scale of FLAN 137B makes it costly to serve.
Revised: Finally, the scale of FLAN 137B makes it costly to serve.

Original: Future work on instruction tuning could include gathering/generating even more task clusters for ﬁnetuning, cross-lingual experiments, using FLAN to generate data for training downstream classiﬁers, and using ﬁnetuning to improve model behavior with respect to bias and fairness (Solaiman & Dennison, 2021).
Revised: Future work on instruction tuning could include gathering/generating even more task clusters for ﬁnetuning, cross-lingual experiments, using FLAN to generate data for training downstream classiﬁers, and using ﬁnetuning to improve model behavior with respect to bias and fairness (Solaiman & Dennison, 2021).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: This paper has explored a simple method for improving the ability of language models at scale to perform zero-shot tasks based purely on instructions.
Revised: This paper has explored a simple method for improving the ability of language models at scale to perform zero-shot tasks based purely on instructions.

Original: Our instruction-tuned model, FLAN, compares favorably against GPT-3 and signals the potential ability for language models at scale to follow instructions.
Revised: Our instruction-tuned model, FLAN, compares favorably against GPT-3 and signals the potential ability for language models at scale to follow instructions.

Original: We hope that our paper will spur further research on instructions-based NLP, zero-shot learning, and using labeled data to improve large language models.-beta
Revised: We hope that our paper will spur further research on instructions-based NLP, zero-shot learning, and using labeled data to improve large language models.

Original: E THICAL C ONSIDERATIONS
Revised: E THICAL C ONSIDERATIONS

Original: This work uses language models, for which the risks and potential harms are discussed in Bender &
Revised: This work uses language models, for which the risks and potential harms are discussed in Bender &

Original: Koller (2020), Brown et al.
Revised: Koller (2020), Brown et al.

Original: (2020), Bender et al.
Revised: (2020), Bender et al.

Original: (2021), Patterson et al.
Revised: (2021), Patterson et al.

Original: , (2021) and others.
Revised: , (2021), and others.

Original: As our contribution in this paper is not a pretrained language model itself but rather an empirical study of how instruction tuning affects the zero-shot performance of a language model on unseen tasks, we additionally highlight two relevant ethical considerations.
Revised: As our contribution in this paper is not a pretrained language model itself but rather an empirical study of how instruction tuning affects the zero-shot performance of a language model on unseen tasks, we additionally highlight two relevant ethical considerations.

Original: First, labeled datasets such as those we use for ﬁnetuning can contain undesirable biases, and these biases can be propagated into zero-shot applications of the model on downstream tasks.
Revised: First, labeled datasets such as those we use for ﬁnetuning can contain undesirable biases, and these biases can be propagated into zero-shot applications of the model on downstream tasks.

Original: And second, instruction-tuned models can potentially require less data and expertise to use; such lower barriers to access could increase both the beneﬁts and associated risks of such models
Revised: And second, instruction-tuned models can potentially require less data and expertise to use; such lower barriers to access could increase both the beneﬁts and associated risks of such models.

Original: A A DDITIONAL R ESULTS A. C OMMONSENSE R EASONING & C OREFERENCE R ESOLUTION
Revised: E NVIRONMENTAL C ONSIDERATIONS

Original: Results for ﬁve commonsense reasoning datasets are shown in Table 4.
Revised: We use the same pretrained language models as Austin et al. The energy cost and carbon footprint for the pretrained models were 451 MWh and 26 tCO2e, respectively. The additional instruction tuning gradient-steps for ﬁnetuning FLAN is less than 2% of the number of pretraining steps, and so the estimated additional energy cost is comparatively smaller. A UTHOR C ONTRIBUTIONS

Original: FLAN outperforms GPT-3 on
Revised: Maarten Bosma conceived the original idea and implemented the ﬁrst version of FLAN. Vincent Zhao prototyped the training and evaluation pipelines, as well as rank classiﬁcation. Kelvin Guu proposed and implemented the idea of task clusters and evaluation using inter-cluster splits. Jason Wei, Maarten

Original: StoryCloze (Mostafazadeh et al., 2016) and has comparable performance on CoPA (Roemmele et al., 2011) and PiQA (Bisk et al., 2020).
Revised: Bosma, Vincent Zhao, and Adams Wei Yu implemented the NLP tasks. Jason Wei, Vincent Zhao, and Adams Wei Yu conducted and managed most of the experiments.

Original: For HellaSwag (Zellers et al., 2019) and ReCoRD (Zhang et al., 2018), however, both Base LM and FLAN performs well below GPT-3.
Revised: Jason Wei designed and ran the ablation studies. Jason Wei, Maarten Bosma, and Quoc V. Le wrote most of the paper. Jason Wei,

Original: Table 4 also shows results for two coreference resolution datasets.
Revised: Maarten Bosma, and Nan Du obtained the zero and few-shot baselines.

Original: FLAN with the best dev template outperforms zero-shot GPTon Winogrande (Sakaguchi et al., 2020).
Revised: Vincent Zhao and Kelvin Guu designed, implemented, and conducted the few-shot FLAN experiments.

Original: For WSC273 (Levesque et al., 2012), both Base LM and FLAN are well below GPT-3.
Revised: Maarten Bosma and Jason Wei ran the data contamination analysis.

Original: Overall, FLAN is comparable with Base LM (with the exception of ReCoRD, which has the longest continuations of all language modeling tasks).
Revised: Brian Lester ran the prompt tuning experiments. Quoc V. Le and Andrew M. Dai advised, provided high-level guidance, and helped edit the paper. A CKNOWLEDGEMENTS

Original: One observation is that HellaSwag, PiQA, ReCoRD, WSC273, and Winogrande are directly formulated in the original pretraining objective of language modeling. Hence, the result that FLAN does not outperform Base LM or GPT-3 is consistent with the intuition that gains from FLAN will be marginal when instructions are not crucial for describing the given task.
Revised: We thank Ed Chi, Slav Petrov, Dan Garrette, Ruibo Liu, and Clara Meister for providing feedback on our manuscript. We thank Adam Roberts, Liam Fedus, Hyung Won Chung, and Noam Shazeer for helping debug some of our models. We thank Ellie Pavlick for feedback on the study design during the middle stages of the project.

Original: Moreover, we note a further limitation with FLAN for these ﬁve language modeling tasks, including options actually hurts performance, and so the reported results are for rank classiﬁcation without options.
Revised: We thank Daniel De Freitas Adiwardana for helping initiate the project, large language model advising, and giving us access to some computational resources. Finally, we thank the team involved in pretraining LaMDA-PT: Daniel De Freitas Adiwardana, Noam

Original: [Table]
Revised: Shazeer, Yanping Huang, Dmitry Lepikhin, Dehao Chen, Yuanzhong Xu and Zhifeng Chen.

Original: A. A DDITIONAL D ATASETS
Revised: A A DDITIONAL R ESULTS

Original: The main paper and Appendix A.1 focused on datasets for which GPT-3 resultsare available for comparison; here we report additional results on datasets for which comparisons with GPT-3 were not available.
Revised: This section shows the full results for all datasets we evaluate.

Original: Results for eight NLU task clusters are shown in Table 5 on the next page [Table] and the results for translation and struct to text are shown in Table 6 on the page after that.
Revised: Results for translation and struct to text are shown in Table 1, and the results for eight NLU task clusters are shown in Table 2.

Original: We show FLAN’s performance using the best of up to ten instruction templates as well as the template with the best performance on the dev set.
Revised: We show FLAN’s performance using the best of up to ten instruction templates as well as the template with the best performance on the dev set.

Original: For Base LM, we use the templates from Brown et al.
Revised: For LaMDA-PT, we use the templates from Brown et al.

Original: (2020), which were optimized for GPT-3, without performing any prompt engineering to optimize them on our model.
Revised: (2020), which were optimized for GPT-3, without performing any prompt engineering to optimize them on our model.

Original: For simplicity, we use greedy search for all generative tasks (compared with beam search used in Brown et al.
Revised: For simplicity, we use greedy search for all generative tasks (compared with beam search used in Brown et al.

Original: Unlike GPT-3, which chooses the number of few-shot exemplars k via best dev set performance, for few-shot Base LM we choose the highest k that ﬁts in the context length of 1024 tokens, from k ∈ { 1 , 3 , 5 , 10 } .
Revised: Unlike GPT-3, which chooses the number of few-shot exemplars k via best dev set performance, for few-shot LaMDA-PT we choose the highest k that ﬁts in the context length of 1024 tokens, from k ∈ { 1 , 3 , 5 , 10 } .

Original: For DROP (Dua et al., 2019) and SQuADv2 (Rajpurkar et al., 2018), based on email correspondence with Brown et al.
Revised: For DROP (Dua et al., 2019) and SQuADv2 (Rajpurkar et al., 2018), based on email correspondence with Brown et al.

Original: (2020), their deﬁnition of zero-shot differs from ours in that they actually use exemplars, but only from the same passage as the inference question (each passage has more than one question).
Revised: (2020), their deﬁnition of zero-shot differs from ours in that they actually use exemplars, but only from the same passage as the inference question (each passage has more than one question).

Original: Hence, GPT-3 zero-shot results are not directly comparable with ours for DROP and
Revised: Hence, GPT-3 zero-shot results are not directly comparable with ours for DROP and

Original: SQuADv2.
Revised: SQuADv2.

Original: We mark these results using the † symbol.
Revised: We mark these results using the † symbol.

Original: Moreover, it is unclear how to parse the end of an answer for these two datasets, and so we use curly bracket delimiters { and } , where we expect } to indicate the end of the answer.
Revised: Moreover, it is unclear how to parse the end of an answer for these two datasets, and so we use curly bracket delimiters { and } , where we expect } to indicate the end of the answer.

Revised: For struct to text, reported T5/mT5 results are from the GEM benchmark paper (Gehrmann et al., 2021), though we do not report their results for DART (through correspondence with authors, we conﬁrmed that their results for DART were incorrect).

Revised: Though we use a summarization task cluster during instruction tuning, we leave evaluation of summarization for future work, as the mean input of most summarization datasets exceeds FLAN’s input length of 1024 tokens

Original: [Table]
Revised: [Table]

Original: WSC273 (Levesque et al., 2012) does not have training or validation sets, and so we do not compute few-shot results for FLAN.

Original: For Trivia QA (TQA), we report exact match (EM) on both the wikipedia subset of the dev set to compare with GPT-3, as well as the full TFDS dev set.

Original: [Table]
Revised: [Table]

Original: For struct to text, reported T5/mT5 results are from the GEM benchmark paper (Gehrmann et al., 2021), though we do not report their results for DART (through correspondence with authors, we conﬁrmed that their results for DART were incorrect). Though we use a summarization task cluster during instruction tuning, we leave evaluation of summarization for future work, as themean input of most summarization datasets exceeds FLAN’s input length of 1024 tokens.
Revised: see contamination (Appendix (Levesque et al., 2012) does not have training or sets, and so we do not compute few-shot for FLAN. Trivia QA (TQA), we report exact match (EM) on both the wikipedia of the to compare with GPT- as the

Original: B F URTHER A BLATION S TUDIES  B. D ATASETS PER T ASK C LUSTER & T EMPLATES PER D ATASET
Revised: B F URTHER A BLATION S TUDIES AND A NALYSIS B. D ATASETS PER T ASK C LUSTER & T EMPLATES PER D ATASET

Original: Our primary hypothesis is that instruction tuning on a diverse set of tasks improves performance on unseen tasks.
Revised: Our primary hypothesis is that instruction tuning on a diverse set of tasks improves performance on unseen tasks.

Original: §4.1 showed that adding more task clusters improves performance; here, we further explore whether adding additional datasets improves performance when the number of task clusters is held constant.
Revised: §4.1 showed that adding more task clusters improves performance; here, we further explore whether adding additional datasets improves performance when the number of task clusters is held constant.

Original: We use the same split as in §4.1, where the NLI, commonsense reasoning, and closed-book QA clusters are held-out, and seven other task clusters remain for instruction tuning.
Revised: We use the same split as in §4.1, where the NLI, commonsense reasoning, and closed-book QA clusters are held-out, and seven other task clusters remain for instruction tuning.

Original: For these seven task clusters, we instruction tune models using just one dataset per task cluster and using four datasets per task cluster (for task clusters that did not have four tasks, we just used all available tasks).
Revised: For these seven task clusters, we instruction tune models using just one dataset per task cluster and using four datasets per task cluster (for task clusters that did not have four tasks, we just used all available tasks).

Original: In addition, we simultaneously explore the role of the number of instruction templates per dataset; as mentioned in §2.1, for each dataset we manually composed ten instructional templates for instruction tuning.
Revised: In addition, we simultaneously explore the role of the number of instruction templates per dataset; as mentioned in §2.1, for each dataset we manually composed ten instructional templates for instruction tuning.

Original: Here, we instruction tune models using 1, 4, and 10 templates per dataset.
Revised: Here, we instruction tune models using 1, 4, and 10 templates per dataset.

Original: Figure 8 shows these results.
Revised: Figure 11 shows these results.

Original: Using more datasets per cluster improved performance by almost 10% on average across the three held-out clusters.
Revised: Using more datasets per cluster improved performance by almost 10% on average across the three held-out clusters.

Original: Using more templates per dataset, however, had a comparatively negligible effect on performance when there was one task per cluster, which disappeared when there were four tasks per cluster.
Revised: Using more templates per dataset, however, had a comparatively negligible effect on performance when there was one task per cluster, which disappeared when there were four tasks per cluster.

Original: The small effect of templates is striking given our original motivation that composing ten templates per task would mitigate overﬁtting to any particular template.
Revised: The small effect of templates is striking given our original motivation that composing ten templates per task would mitigate overﬁtting to any particular template.

Original: This results serves to underscore, however, the unpredictability of ﬁnetuning large language models, as one hypothesis is that models at such scale do not easily overﬁt to a ﬁnetuning single task.small disappeared
Revised: This results serves to underscore, however, the unpredictability of ﬁnetuning large language models, as one hypothesis is that models at such scale do not easily overﬁt to a ﬁnetuning single task.

Original: B. R OLEOF INSTRUCTIONS DURING FINETUNING
Revised: B. R OLE

Original: Instruction tuning ﬁnetunes a model on a collection of tasks phrased as instructions, with the goal of training the model to follow them. While instructions must be used for zero-shot inference to indicate which task to perform, one possibility is that the performance gains come entirely from multi-task ﬁnetuning and that the model could perform just as well with missing prompts during ﬁnetuning. To tease apart this point, we performed an ablation experiment where the instructional prompts were removed during ﬁnetuning.

Original: We considered the following two setups without instructions: No template. Only the inputs and outputs were given to the model, as would be the case for a task-speciﬁc model. E.g., for translation to French, the input would be “the dog runs” and the output would be “le chien court” .
Revised: INSTRUCTIONS

Original: The input does not distinguish among tasks during multi-task ﬁnetuning.

Original: Task/dataset name only.

Original: In this setup, each input is prepended with the name of the task and the name of the dataset.

Original: E.g., for translation to French, the input would be “[Translation: WMT’14 to French]

Original: The dog runs . We compare these two setups to FLAN’s ﬁneuning procedure, which used natural instructions (e.g., “‘The dog runs.’ Translate this sentence to French.” ). We did these ablation experiments for the four held-out clusters used in the main paper, with the exact same experimental setup as FLAN results in §3.
Revised: The per-cluster results for the ablation study from §4.3 are shown in Table 3.

Original: For the no template setup, we used the FLAN instructions during zero-shot inference (because if we used no template, the model would not know what task to perform).

Original: For models ﬁnetuned on task/dataset name only, we report zero-shot inference performance for FLAN instructions as well as using the task/dataset name.

Original: When evaluating using natural instructions, we follow the same procedure as for FLAN and report all ten templates and report the average.

Original: These results are shown in Table 7 below.

Original: Finetuning with no templates has very poor performance, which is expected since the model is not able to distinguish between tasks during ﬁnetuning.

Original: Finetuning with the task/dataset name improves performance noticeably over no template (using task/dataset name for inference versus natural instructions for inference performed about the same in aggregate), but is still almost eight points behind FLAN on average across the four task clusters.

Original: This result indicates that training with instructions is crucial for zero-shot performance on unseen tasks (and is consistent with Min et al.

Original: (2021), which showed that instructions even help for few-shot ﬁnetuning).

Original: [Table]

Original: C F URTHER A NALYSIS : I NSTRUCTION T UNING F ACILITATES P ROMPT T UNING
Revised: B. F URTHER A NALYSIS : I NSTRUCTION T UNING F ACILITATES P ROMPT T UNING

Original: As we’ve seen that instruction tuning improves the ability of a model to respond to instructions, it follows that, if FLAN is indeed more amenable to performing NLP tasks, then it should also achieve better performance when performing inference using soft prompts, represented by prepended continuous variables optimized via prompt tuning (Li & Liang, 2021; Lester et al., 2021).

Original: As further analysis, we train continuous prompts for each of the SuperGLUE (Wang et al., 2019a) tasks in accordance with the cluster splits from §2.2 such that when prompt-tuning on task T , no tasks in the same cluster as T were seen during instruction tuning.

Original: Our prompt tuning setup follows the procedure of Lester et al.

Original: (2021) except that we use a prompt length of 10, weight decay of 1e-

Original: [Table]

Original: and did not use dropout on the attention scores; we found in preliminary experiments that these changes improved the performance of Base LM.

Original: Table 8 shows the results of these prompt tuning experiments for both using a fully-supervised training set and in a low-resource setting with only 32 training examples.

Original: We see that in all scenarios, prompt tuning works better with FLAN than Base LM.

Original: In many cases, especially for the low-resource setting, prompt tuning on FLAN even achieves more than 10% improvement over prompt tuning on the Base LM.

Original: This result exempliﬁes in another way how instruction tuning can result in a checkpoint that is more desirable for performing NLP tasks

Original: [Table]

Original: P ROMPT

Original: [Table] UNING A NALYSIS [Table] from the same cluster as that dataset were seen during instruction tuning.
Revised: The per-dataset results for the analysis in §4.5 are given in Table 4.

Original: As the above tasks are all classiﬁcation, further work in this direction might include tasks such as summarization or question answering, or try to ﬁnetune the model using the supervised datasets.
Revised: As the above tasks are all classiﬁcation, further work in this direction might include tasks such as summarization or question answering, or try to ﬁnetune the model using the supervised datasets.

Original: D D ATA C ONTAMINATION A NALYSIS
Revised: C D ATA C ONTAMINATION A NALYSIS

Original: One reasonable concern is that since the pretraining corpus of FLAN has more than 2 trillion tokens, it is possible that examples from a given evaluation dataset may have already been seen verbatim by the model during pre-training, hence inﬂating the performance of our purported zero-shot model.
Revised: One reasonable concern is that since the pretraining corpus of FLAN has more than 2 trillion tokens, it is possible that examples from a given evaluation dataset may have already been seen verbatim by the model during pre-training, hence inﬂating the performance of our purported zero-shot model.

Original: To this end, like GPT-3 (Brown et al., 2020), we perform post-hoc data contamination analysis to  investigate whether the performance of the model is in fact inﬂated by evaluating on examples that occurred in the pretraining dataset.
Revised: To this end, like GPT-3 (Brown et al., 2020), we perform post-hoc data contamination analysis to [Table] [Table] investigate whether the performance of the model is in fact inﬂated by evaluating on examples that occurred in the pretraining dataset.

Original: Our data contamination procedure follows the setup of Brown et al.
Revised: Our data contamination procedure follows the setup of Brown et al.

Original: (2020), which, for each benchmark, produces a “clean” version that removes all potentially leaked examples, deﬁned as examples for which any n -gram ( n varies per dataset but is roughly 13) overlapped with anything in the pretraining corpus.
Revised: (2020), which, for each benchmark, produces a “clean” version that removes all potentially leaked examples, deﬁned as examples for which any n -gram ( n varies per dataset but is roughly 13) overlapped with anything in the pretraining corpus.

Original: We use the same n per dataset as Brown et al.
Revised: We use the same n per dataset as Brown et al.

Original: (2020), only differing in that we use the SentencePiece tokenizer (Kudo & Richardson, 2018) instead of splitting on spaces.
Revised: and also split on spaces.

Original: We then evaluate our model on this clean subset, comparing against model performance on the original dataset (clean + dirty).
Revised: We then evaluate our model on this clean subset, comparing against model performance on the original dataset (clean + dirty).

Original: Lower performance on the clean subset would suggest that data contamination leads to inﬂated results.
Revised: Lower performance on the clean subset would suggest that data contamination leads to inﬂated results.

Original: Figure 14 summarizes these results, with the exact numbers given in Table 9.
Revised: Figure 12 summarizes these results, with the exact numbers given in Table 5.

Original: We see several trends very similar to those in the GPT-3 paper: (1) many datasets had a substantial number of examples that overlapped with the pretraining data, (2) across all datasets, we do not see a correlation that evaluating on clean data does worse than evaluating on the total dataset, and (3) as datasets had fewer clean examples, there was higher variance in the percent change in performance (likely due to a smaller number of clean examples).
Revised: We see several trends very similar to those in the GPT-3 paper: (1) many datasets had a substantial number of examples that overlapped with the pretraining data, (2) across all datasets, we do not see a correlation that evaluating on clean data does worse than evaluating on the total dataset, and (3) as datasets had fewer clean examples, there was higher variance in the percent change in performance (likely due to a smaller number of clean examples).

Original: [Figure]

Original: Like GPT-3, we also found that DROP and SQuADv2 had almost total overlap with the pretraining data.
Revised: Like GPT-3, we also found that DROP and SQuADv2 had almost total overlap with the pretraining data.

Original: We follow their procedure of manually inspecting the data, and ﬁnd that most overlapping ngrams were only in the contexts of examples (99.6% for DROP and 97.2% for SQuADv2).
Revised: We follow their procedure of manually inspecting the data, and ﬁnd that most overlapping ngrams were only in the contexts of examples (99.6% for DROP and 97.2% for SQuADv2).

Original: Overlaps never occurred in both the question and answer for DROP, and only occurred for both the question and answer for SQuADv2 in 5 of the 11,153 evaluation examples.
Revised: Overlaps never occurred in both the question and answer for DROP, and only occurred for both the question and answer for SQuADv2 in 5 of the 11,153 evaluation examples.

Original: Hence, for these two datasets, the  model gains only background information and cannot memorize the answer to any speciﬁc questions (aside from the ﬁve examples in SQuADv2).
Revised: Hence, for these two datasets, the [Figure] model gains only background information and cannot memorize the answer to any speciﬁc questions (aside from the ﬁve examples in SQuADv2).

Original: ANLI R1 and R2 (Nie et al., 2020) also had almost complete data contamination, to a much higher degree than GPT-3.
Revised: ANLI R1 and R2 (Nie et al., 2020) also had almost complete data contamination, to a much higher degree than GPT-3.

Original: Upon further inspection, we see that most overlaps occur in example contexts and not hypotheses (97.3% for ANLI R1 and 98.2% for ANLI R2).
Revised: Upon further inspection, we see that most overlaps occur in example contexts and not hypotheses (97.3% for ANLI R1 and 98.2% for ANLI R2).

Original: As ANLI R1 and R2 are based entirely from Wikipedia examples (R3 is not), we posit that this higher degree of contamination in our pretraining dataset compared with GPT-3’s is potentially due to using a more-recent version of Wikipedia that includes the contexts used in ANLI R1 and R2 (which were collected in 2019).
Revised: As ANLI R1 and R2 are based entirely from Wikipedia examples (R3 is not), we posit that this higher degree of contamination in our pretraining dataset compared with GPT-3’s is potentially due to using a more-recent version of Wikipedia that includes the contexts used in ANLI R1 and R2 (which were collected in 2019).

Original: Because seeing a particular context in pretraining does not help with the NLI task given a new, unseen sentence, we think it is unlikely that these overlaps affected performance on the two datasets.
Revised: Because seeing a particular context in pretraining does not help with the NLI task given a new, unseen sentence, we think it is unlikely that these overlaps affected performance on the two datasets.

Original: Of the remaining datasets, only ReCoRD and PIQA had a clean subset performance that was lower than the overall evaluation set performance by more than 1%.
Revised: Of the remaining datasets, only ReCoRD and PIQA had a clean subset performance that was lower than the overall evaluation set performance by more than 1%.

Original: These two datasets are language modeling (i.e., “what’s the best continuation of this sentence?”), and so it is more likely compared with previous tasks that seeing a complete sentence in the pretraining data could help the model predict the right answer in downstream evaluations.
Revised: These two datasets are language modeling (i.e., “what’s the best continuation of this sentence?”), and so it is more likely compared with previous tasks that seeing a complete sentence in the pretraining data could help the model predict the right answer in downstream evaluations.

Original: For PIQA, both the goal and solution had overlaps in 93 of the 1,838 evaluation examples, and for ReCoRD, the query had overlaps in 2,320 of the 10,000 training examples.
Revised: For PIQA, both the goal and solution had overlaps in 93 of the 1,838 evaluation examples, and for ReCoRD, the query had overlaps in 2,320 of the 10,000 training examples.

Original: We hence mark these results with an asterisk ∗ in Table 4 and Table 5.
Revised: We hence mark these results with an asterisk ∗ in Table 2.

Original: Brown et al.
Revised: Brown et al.

Original: (2020) also reported substantial contamination rates for these two datasets (61% dirty for ReCoRD and 29% for PIQA), and also mark PIQA with an asterisk.
Revised: (2020) also reported substantial contamination rates for these two datasets (61% dirty for ReCoRD and 29% for PIQA), and also mark PIQA with an asterisk.

Original: As this overlap analysis follows that performed in Brown et al.
Revised: As this overlap analysis follows that performed in Brown et al.

Original: (2020), we reiterate the same caveats: the conservative nature of our n -gram matching procedure likely introduces additional false positives; there are no guarantees that the clean subset is drawn from the same distribution as the overall subset; and, accurately detecting test contamination is a relatively new research area without established best practices.
Revised: (2020), we reiterate the same caveats: the conservative nature of our n -gram matching procedure likely introduces additional false positives; there are no guarantees that the clean subset is drawn from the same distribution as the overall subset; and, accurately detecting test contamination is a relatively new research area without established best practices.

Original: Moreover, as our pretraining corpus is almost ﬁve times larger than that used for GPT- (which was 500B tokens), it is possible that there are more false positives in detecting dirty data.
Revised: Moreover, as our pretraining corpus is almost ﬁve times larger than that used for GPT- (which was 500B tokens), it is possible that there are more false positives in detecting dirty data.

Original: [Table]
Revised: [Table]

Original: E E XTENDED R ELATED W ORK E. L ANGUAGE M ODELS AND M ULTI - TASK L EARNING
Revised: D E XTENDED R ELATED W ORK D. L ANGUAGE M ODELS AND M ULTI - TASK L EARNING

Original: Our work is broadly inspired by a long line of prior work on language models for NLP applications (Dai & Le, 2015;
Revised: Our work is broadly inspired by a long line of prior work on language models for NLP applications (Dai & Le, 2015;

Original: Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; 2019, inter alia ).
Revised: Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; 2019, inter alia ).

Original: Instruction tuning can be seen as a formulation of multitask learning (MTL), which is an established area within deep learning (Collobert et al., 2011;
Revised: Instruction tuning can be seen as a formulation of multitask learning (MTL), which is an established area within deep learning (Collobert et al., 2011;

Original: Luong et al., 2016; Ruder, 2017; Velay & Daniel, 2018; Clark et al., 2019b; Liu et al., 2019b, inter alia )—see Worsham & Kalita (2020) for a recent survey on MTL for NLP.
Revised: Luong et al., 2016; Ruder, 2017; Velay & Daniel, 2018; Clark et al., 2019b; Liu et al., 2019b, inter alia )—see Worsham & Kalita (2020) for a recent survey on MTL for NLP.

Original: Differing from prior MTL work which focuses on performance improvements across training tasks (Raffel et al., 2020; Aghajanyan et al., 2021) or to new domains (Axelrod et al., 2011), our work is motivated by improving zero-shot generalization to tasks that were not seen in training.
Revised: Differing from prior MTL work which focuses on performance improvements across training tasks (Raffel et al., 2020; Aghajanyan et al., 2021) or to new domains (Axelrod et al., 2011), our work is motivated by improving zero-shot generalization to tasks that were not seen in training.

Original: E. Z ERO -S HOT L EARNING AND M ETA -L EARNING
Revised: D. Z ERO -S HOT L EARNING AND M ETA -L EARNING

Original: Our work also falls in the well-established category of zero-shot learning, which has historically been used to refer to classifying instances among a set of unseen categories (Lampert et al., 2009;
Revised: Our work also falls in the well-established category of zero-shot learning, which has historically been used to refer to classifying instances among a set of unseen categories (Lampert et al., 2009;

Original: Romera-Paredes & Torr, 2015; Srivastava et al., 2018;
Revised: Romera-Paredes & Torr, 2015; Srivastava et al., 2018;

Original: Yin et al., 2019, inter alia ).
Revised: Yin et al., 2019, inter alia ).

Original: In NLP, zero-shot learning work also includes translating between unseen language pairs (Johnson et al., 2017; Pham et al., 2019), language modeling on unseen languages (Lauscher et al., 2020), as well as various NLP applications (Liu et al., 2019a; Corazza et al., 2020; Wang et al.,2021; Reif et al., 2021).
Revised: In NLP, zero-shot learning work also includes translating between unseen language pairs (Johnson et al., 2017; Pham et al., 2019), language modeling on unseen languages (Lauscher et al., 2020), as well as various NLP applications (Liu et al., 2019a; Corazza et al., 2020; Wang et al., 2021).

Original: Most recently, the emergent ability of language models (Brown et al., 2020) has led to increased interest in how models generalize to unseen tasks, the deﬁnition of zero-shot learning used in our paper.
Revised: Most recently, the emergent ability of language models (Brown et al., 2020) has led to increased interest in how models generalize to unseen tasks, the deﬁnition of zero-shot learning used in our paper.

Original: In addition, meta-learning (Finn et al., 2017; Vanschoren, 2018, inter alia ) also broadly tries to train models that adapt quickly to unseen tasks, typically based on a few examples.
Revised: In addition, meta-learning (Finn et al., 2017; Vanschoren, 2018, inter alia ) also broadly tries to train models that adapt quickly to unseen tasks, typically based on a few examples.

Original: E. P ROMPTING
Revised: D. P ROMPTING

Original: Instruction tuning leverages the intuition that language models at scale contain substantial world knowledge and can perform a range of NLP tasks (Brown et al., 2020, see also Bommasani et al.
Revised: Instruction tuning leverages the intuition that language models at scale contain substantial world knowledge and can perform a range of NLP tasks (Brown et al., 2020, see also Bommasani et al.

Original: Another line of work that shares this goal prompts models with continuous inputs optimized via backpropagation to substantially improve performance (Li & Liang, 2021; Lester et al., 2021; Qin & Eisner, 2021).
Revised: Another line of work that shares this goal prompts models with continuous inputs optimized via backpropagation to substantially improve performance (Li & Liang, 2021; Lester et al., 2021; Qin & Eisner, 2021), as well as work that prompts models to produce specialized outputs (Wei et al., 2022).

Original: Although the success of these approaches depends heavily on model scale (Lester et al., 2021), for which large models can be costly to serve, the ability of a single large model to perform many tasks slightly eases this burden.
Revised: Although the success of these approaches depends heavily on model scale (Lester et al., 2021), for which large models can be costly to serve, the ability of a single large model to perform many tasks slightly eases this burden.

Original: As shown by our experiments in Appendix C, prompt tuning is an orthogonal method for which instruction tuning can additionally improve performance.
Revised: As shown by our experiments in §4.5, prompt tuning is an orthogonal method for which instruction tuning can additionally improve performance.

Revised: Reif et al.

Revised: (2021) is similar to our work in that they also use related tasks to improve zero-shot learning, though they differ by only using related tasks in the context (and not ﬁnetuning), and focus on the application of text style transfer.

Original: Our work shares similar motivations with prompting in that we use inference-time text interactions to prompt a single model, without creating separate checkpoints for each task.
Revised: Our work shares similar motivations with prompting in that we use inference-time text interactions to prompt a single model, without creating separate checkpoints for each task.

Original: Whereas prompting work such as GPT-3 uses prompt engineering to write prompts that intentionally mimic text that is likely to be seen during pretraining (e.g., for MultiRC GPT-3 tries a prompt that mimics a test with an answer key), we hope that ﬁnetuning models to respond to natural language instructions instead of completing a sentence will make such large models more accessible to non-technical users.
Revised: Whereas prompting work such as GPT-3 uses prompt engineering to write prompts that intentionally mimic text that is likely to be seen during pretraining (e.g., for MultiRC GPT-3 tries a prompt that mimics a test with an answer key), we hope that ﬁnetuning models to respond to natural language instructions instead of completing a sentence will make such large models more accessible to non-technical users.

Original: E. F INETUNING L ARGE L ANGUAGE M ODELS
Revised: D. F INETUNING L ARGE L ANGUAGE M ODELS

Original: Finetuning pretrained language models is a well-established method in NLP, with much of the work so far occurring on models in the range of 100M to 10B parameters (Dai & Le, 2015; Devlin et al., 2019; Raffel et al., 2020; Lewis et al., 2020, inter alia ).
Revised: Finetuning pretrained language models is a well-established method in NLP, with much of the work so far occurring on models in the range of 100M to 10B parameters (Dai & Le, 2015; Devlin et al., 2019; Raffel et al., 2020; Lewis et al., 2020, inter alia ).

Original: For models of O(100B) parameters, recent work has ﬁnetuned task-speciﬁc models for program synthesis (Austin et al., 2021; Chen et al., 2021), summarization (Wu et al., 2021), as well as improved bias and fairness behavior (Solaiman & Dennison, 2021).
Revised: For models of O(100B) parameters, recent work has ﬁnetuned task-speciﬁc models for program synthesis (Austin et al., 2021; Chen et al., 2021), summarization (Wu et al., 2021), as well as improved bias and fairness behavior (Solaiman & Dennison, 2021).

Original: In addition to the traditional “dense” models, sparse mixture of experts (MoE) models of up to more than 1T parameters have been trained and ﬁnetuned (Lepikhin et al., 2020; Fedus et al., 2021).
Revised: In addition to the traditional “dense” models, sparse mixture of experts (MoE) models of up to more than 1T parameters have been trained and ﬁnetuned (Lepikhin et al., 2020; Fedus et al., 2021).

Original: Compared with this prior work that ﬁnetunes and evaluates on the same downstream task, our setup studies the effect of instruction tuning on ability to perform unseen tasks.
Revised: Compared with this prior work that ﬁnetunes and evaluates on the same downstream task, our setup studies the effect of instruction tuning on ability to perform unseen tasks.

Original: E. M ULTI - TASK Q UESTION A NSWERING
Revised: D. M ULTI - TASK Q UESTION A NSWERING

Original: The instructions we use for instruction tuning are similar to QA-based task formulation research, which aims to unify NLP tasks by casting them as question-answering over a context.
Revised: The instructions we use for instruction tuning are similar to QA-based task formulation research, which aims to unify NLP tasks by casting them as question-answering over a context.

Original: For instance,
Revised: For instance,

Original: McCann et al.
Revised: McCann et al.

Original: (2018) cast ten NLP tasks as QA and train a model on a collection of tasks formulated with natural language prompts; they report transfer learning gains on ﬁnetuning tasks as well as zero-shot domain adaptation results on SNLI (Bowman et al., 2015) and Amazon/Yelp Reviews (Kotzias et al., 2015).
Revised: (2018) cast ten NLP tasks as QA and train a model on a collection of tasks formulated with natural language prompts; they report transfer learning gains on ﬁnetuning tasks as well as zero-shot domain adaptation results on SNLI (Bowman et al., 2015) and Amazon/Yelp Reviews (Kotzias et al., 2015).

Original: While McCann et al.
Revised: While McCann et al.

Original: (2018) does not leverage unsupervised pre-training and only reports zero-shot transfer to unseen domains, our work uses a pretrained LM and focuses on zero-shot performance on unseen task clusters.
Revised: (2018) does not leverage unsupervised pre-training and only reports zero-shot transfer to unseen domains, our work uses a pretrained LM and focuses on zero-shot performance on unseen task clusters.

Original: UniﬁedQA (Khashabi et al., 2020) shows similar transfer learning gains as McCann et al.
Revised: UniﬁedQA (Khashabi et al., 2020) shows similar transfer learning gains as McCann et al.

Original: (2018) across 20 datasets and reports good generalization to unseen tasks across four types of QA.
Revised: (2018) across 20 datasets and reports good generalization to unseen tasks across four types of QA.

Original: Focusing on binary text classiﬁcation, Zhong et al.
Revised: Focusing on binary text classiﬁcation, Zhong et al.

Original: (2021) ﬁnetune T5-770M on 43 tasks phrased as yes/no questions and study the zero-shot performance on unseen tasks.
Revised: (2021) ﬁnetune T5-770M on 43 tasks phrased as yes/no questions and study the zero-shot performance on unseen tasks.

Original: In comparison, our paper is much larger in scope, empirically demonstrating the idea on a wide range of tasks with a much larger model.
Revised: In comparison, our paper is much larger in scope, empirically demonstrating the idea on a wide range of tasks with a much larger model.

Original: Other work has used QA-based task formulation for more-targeted applications including semantic role labeling (He et al., 2015), relation extraction (Levy et al., 2017), coreference resolution (Wu et al., 2020) and named entity recognition (Li et al., 2020) as question answering.
Revised: Other work has used QA-based task formulation for more-targeted applications including semantic role labeling (He et al., 2015), relation extraction (Levy et al., 2017), coreference resolution (Wu et al., 2020) and named entity recognition (Li et al., 2020) as question answering.

Original: E. I NSTRUCTIONS -B ASED NLP
Revised: D. I NSTRUCTIONS -B ASED NLP

Original: Recent improvements in the capabilities of language models have led to increased interest in a nascent area of instructions-based NLP (Goldwasser & Roth, 2014, and see McCarthy (1960)).
Revised: Recent improvements in the capabilities of language models have led to increased interest in a nascent area of instructions-based NLP (Goldwasser & Roth, 2014, and see McCarthy (1960)).

Original: Schick &
Revised: Schick &

Original: Schütze (2021) (also see Gao et al., 2021;
Revised: Schütze (2021) (also see Gao et al., 2021;

Original: Tam et al., 2021) use task descriptions in cloze-style phrases to help language models assign soft labels for few-shot and semi-supervised learning, though this line of work ﬁnetunes new checkpoints for each downstream task.
Revised: Tam et al., 2021) use task descriptions in cloze-style phrases to help language models assign soft labels for few-shot and semi-supervised learning, though this line of work ﬁnetunes new checkpoints for each downstream task.

Original: Efrat & Levy (2020) evaluated
Revised: Efrat & Levy (2020) evaluated

Original: GPT-2 (Radford et al., 2019) on simple tasks ranging from retrieving the n th word of a sentence to generating examples for SQuAD, concluding that GPT-2 performs poorly across all tasks.
Revised: GPT-2 (Radford et al., 2019) on simple tasks ranging from retrieving the n th word of a sentence to generating examples for SQuAD, concluding that GPT-2 performs poorly across all tasks.

Original: In terms of the setup of ﬁnetuning on a large number of tasks and evaluating on unseen tasks, two recent papers are similar to ours.
Revised: In terms of the setup of ﬁnetuning on a large number of tasks and evaluating on unseen tasks, two recent papers are similar to ours.

Original: Mishra et al.
Revised: Mishra et al.

Original: (2021) ﬁnetune BART (Lewis et al., 2020) using instructions and few-shot examples for tasks such as question answering, text classiﬁcation, and text modiﬁcation, and ﬁnd that this few-shot ﬁnetuning with instructions improves performance on unseen tasks.
Revised: (2021) ﬁnetune BART (Lewis et al., 2020) using instructions and few-shot examples for tasks such as question answering, text classiﬁcation, and text modiﬁcation, and ﬁnd that this few-shot ﬁnetuning with instructions improves performance on unseen tasks.

Original: Ye et al.
Revised: Ye et al.

Original: (2021) introduce a setup for cross-task few-shot learning, ﬁnding that multi-task meta-learning using MAML (Finn et al., 2017) improves the few-shot capabilities of BART on unseen downstream tasks.
Revised: (2021) introduce a setup for cross-task few-shot learning, ﬁnding that multi-task meta-learning using MAML (Finn et al., 2017) improves the few-shot capabilities of BART on unseen downstream tasks.

Original: Our work differs from these two papers in that we focus on zero-shot learning, for which we observe the crucial importance of model scale (FLAN is 1,000x larger than BART-base).
Revised: Our work differs from these two papers in that we focus on zero-shot learning, for which we observe the crucial importance of model scale (FLAN is 1,000x larger than BART-base).

Original: Perhaps the papers most related to ours are the recent Sanh et al.
Revised: Perhaps the papers most related to ours are the recent Sanh et al.

Original: (2021) and Min et al.
Revised: (2021) and Min et al.

Original: (2021), which were released after our initial preprint.
Revised: (2021), which were released after our initial preprint.

Original: Min et al.
Revised: Min et al.

Original: (2021) ﬁnetunes GPT-2 Large (770M parameters) to be a few-shot learner, which is the same approach as our experiment in Section 4.3.
Revised: (2021) ﬁnetunes GPT-2 Large (770M parameters) to be a few-shot learner, which is the same approach as our experiment in Section 4.3.

Original: Similar to our conclusions, they also observe that including few-shot exemplars and instruction tuning are complementary ways to improve performance.
Revised: Similar to our conclusions, they also observe that including few-shot exemplars and instruction tuning are complementary ways to improve performance.

Original: Sanh et al.
Revised: Sanh et al.

Original: (2021) propose to ﬁnetune T5-11B to respond to prompts, and they also report performance improvements on zero-shot learning.
Revised: (2021) propose to ﬁnetune T5-11B to respond to prompts, and they also report performance improvements on zero-shot learning.

Original: These two papers and our work all study ﬁnetuning with instructions, but, as noted by Min et al.
Revised: These two papers and our work all study ﬁnetuning with instructions, but, as noted by Min et al.

Original: (2021), it is hard to directly compare results, due to differing model sizes, model types (decoder-only vs encoder-decoder), pretraining data, task mixtures, and type of instructions (Sanh et al.
Revised: (2021), it is hard to directly compare results, due to differing model sizes, model types (decoder-only vs encoder-decoder), pretraining data, task mixtures, and type of instructions (Sanh et al.

Original: (2021) say that their instructions are more diverse).
Revised: (2021) say that their instructions are more diverse).

Original: Finally, OpenAI has a beta product called the Instruct Series.
Revised: Finally, OpenAI has a model called InstructGPT (Ouyang et al., 2022).

Original: Though they have not released a research paper on it at this time, based on examples posted on their website, 4 their product appears to have goals similar to those in our work.
Revised: InstructGPT uses human anntations to guide desired model behavior, both via ﬁnetuning and reinforcement learning, ﬁnding that InstructGPT is preferred by human rathers compared with unmodiﬁed GPT-3.

Original: F F REQUENTLY A SKED Q UESTIONS
Revised: E F REQUENTLY A SKED Q UESTIONS

Original: How do the FLAN instructions differ from GPT-3 or T5 prompts?
Revised: How do the FLAN instructions differ from GPT-3 or T5 prompts?

Original: GPT-3 prompting is done in a way such that the prompt looks like data that the model has been pretrained on, and the model ﬁnishes the continuation.
Revised: GPT-3 prompting is done in a way such that the prompt looks like data that the model has been pretrained on, and the model ﬁnishes the continuation.

Original: T5 prompts are mostly just a tag for the dataset, which would not work in the zero-shot setting.
Revised: T5 prompts are mostly just a tag for the dataset, which would not work in the zero-shot setting.

Original: In contrast, the prompts that we use for FLAN are similar to what would be used to ask a human to perform the task.
Revised: In contrast, the prompts that we use for FLAN are similar to what would be used to ask a human to perform the task.

Original: For instance, given an input for an NLI task, these would be the prompts.
Revised: For instance, given an input for an NLI task, these would be the prompts.

Original: T5 prompt: cb hypothesis:
Revised: T5 prompt: cb hypothesis:

Original: At my age you will probably have learnt one lesson.
Revised: At my age you will probably have learnt one lesson.

Original: premise:
Revised: premise:

Original: It’s not certain how many lessons you’ll learn by your thirties.
Revised: It’s not certain how many lessons you’ll learn by your thirties.

Original: GPT-3 prompt:
Revised: GPT-3 prompt:

Original: At my age you will probably have learnt one lesson.
Revised: At my age you will probably have learnt one lesson.

Original: question:
Revised: question:

Original: It’s not certain how many lessons you’ll learn by your thirties.
Revised: It’s not certain how many lessons you’ll learn by your thirties.

Original: true, false, or neither?
Revised: true, false, or neither?

Original: answer:
Revised: answer:

Original: FLAN prompt:
Revised: FLAN prompt:

Original: Premise:
Revised: Premise:

Original: At my age you will probably have learnt one lesson.
Revised: At my age you will probably have learnt one lesson.

Original: Hypothesis:
Revised: Hypothesis:

Original: It’s not certain how many lessons you’ll learn by your thirties.
Revised: It’s not certain how many lessons you’ll learn by your thirties.

Original: Doesthepremise entail the hypothesis?
Revised: Doesthepremise entail the hypothesis?

Original: So because FLAN prompts are formulated as responding to an instruction, they do not work well for pretrained language models without ﬁnetuning.
Revised: So because FLAN prompts are formulated as responding to an instruction, they do not work well for pretrained language models without ﬁnetuning.

Original: Performance was near zero for most generation tasks.
Revised: Performance was near zero for most generation tasks.

Original: For instance, given the input “‘The dog runs.’
Revised: For instance, given the input “‘The dog runs.’

Original: Translate this sentence to French.”
Revised: Translate this sentence to French.”

Original: , Base LM continues with ”The dog runs after the cat” instead of actually translating the sentence.
Revised: , LaMDA-PT continues with ”The dog runs after the cat” instead of actually translating the sentence.

Original: Hence, we used the established GPT-3 prompts for our Base LM baselines.
Revised: Hence, we used the established GPT-3 prompts for our LaMDA-PT baselines.

Original: What are some limitations/failure cases of FLAN?
Revised: What are some limitations/failure cases of FLAN?

Original: While we qualitatively ﬁnd that FLAN responds well to most tasks, it does fail on some simple tasks.
Revised: While we qualitatively ﬁnd that FLAN responds well to most tasks, it does fail on some simple tasks.

Original: For instance, as shown in Figure 19, FLAN fails at the very simple task of returning the second word in a sentence, and also incorrectly translates a question to Danish when asked to answer the question in Danish.
Revised: For instance, as shown in Figure 22, FLAN fails at the very simple task of returning the second word in a sentence, and also incorrectly translates a question to Danish when asked to answer the question in Danish.

Original: Additional limitations include a context length of only 1024 tokens (which is not enough for most summarization tasks), and that the model was mostly trained on English data.
Revised: Additional limitations include a context length of only 1024 tokens (which is not enough for most summarization tasks), and that the model was mostly trained on English data.

Original: Can FLAN be used when large amounts of training data are available?
Revised: Can FLAN be used when large amounts of training data are available?

Original: In this work, we focus on cross-task generalization to zero-shot tasks, but we also believe that instruction tuning could result in positive task transfer among seen tasks, depending on the mixture of tasks (though we leave this for future work).
Revised: In this work, we focus on cross-task generalization to zero-shot tasks, but we also believe that instruction tuning could result in positive task transfer among seen tasks, depending on the mixture of tasks (though we leave this for future work).

Original: In Appendix C, where we apply prompt tuning to the FLAN checkpoint, we see promising results that indicate positive task transfer in a supervised setting.
Revised: In §4.5, where we apply prompt tuning to the FLAN checkpoint, we see promising results that indicate positive task transfer in a supervised setting.

Original: Are the ten unique templates per dataset or per task cluster?
Revised: Are the ten unique templates per dataset or per task cluster?

Original: The ten unique templates are for each dataset and not for a task cluster.
Revised: The ten unique templates are for each dataset and not for a task cluster.

Original: This is because datasets in the same task cluster often differed slightly (e.g., “is this movie review positive” vs “is this yelp review positive” ).
Revised: This is because datasets in the same task cluster often differed slightly (e.g., “is this movie review positive” vs “is this yelp review positive” ).

Original: In Figure 6A, why does the untuned Base LM model see worse performance with more parameters for reading comprehension and sentiment analysis?
Revised: In Figure 7A, why does the untuned LaMDA-PT model see worse performance with more parameters for reading comprehension and sentiment analysis?

Original: For context, Figure 6A is a check of correctness for Figure 6B. Figure 6A conﬁrms that scale improves performance for tasks that were seen during instruction tuning, as expected.
Revised: For context, Figure 7A is a check of correctness for Figure 7B. Figure 7A conﬁrms that scale improves performance for tasks that were seen during instruction tuning, as expected.

Original: The untuned Base LM model performance in Figure 6A is shown just for completeness.
Revised: The untuned LaMDA-PT model performance in Figure 7A is shown just for completeness.

Original: Nonetheless, the fact that scale does not always improve zero-shot performance of untuned Base LM is an interesting artifact.
Revised: Nonetheless, the fact that scale does not always improve zero-shot performance of untuned LaMDAPT is an interesting artifact.

Original: Initially, we were surprised, because Brown et al.
Revised: Initially, we were surprised, because Brown et al.

Original: (2020) shows that scale improves performance across a large number of tasks in aggregate.
Revised: (2020) shows that scale improves performance across a large number of tasks in aggregate.

Original: It turns out that scale does not improve performance for certain tasks.
Revised: It turns out that scale does not improve performance for certain tasks.

Original: This is especially true for zero-shot learning, and we think that this happens to be the case for the reading comprehension and sentiment analysis tasks we evaluate.
Revised: This is especially true for zero-shot learning, and we think that this happens to be the case for the reading comprehension and sentiment analysis tasks we evaluate.

Original: The GPT-3 paper itself similarly reports that zero-shot performance on BoolQ and DROP decreases from 13B to 175B parameters.
Revised: The GPT-3 paper itself similarly reports that zero-shot performance on BoolQ and DROP decreases from 13B to 175B parameters.

Original: The GPT-3 paper does not show results on sentiment analysis, but Holtzman et al.
Revised: The GPT-3 paper does not show results on sentiment analysis, but Holtzman et al.

Original: ﬁnd that zero-shot performance on
Revised: ﬁnd that zero-shot performance on

Original: SST-2 also gets worse from 13B to 175B parameters.
Revised: SST-2 also gets worse from 13B to 175B parameters.

Original: Hence, this artifact is consistent across both
Revised: Hence, this artifact is consistent across both

Original: GPT-3 and the models we use.
Revised: GPT-3 and the models we use.

Original: This artifact is certainly worth further study, but is outside the scope of instruction tuning.
Revised: This artifact is certainly worth further study, but is outside the scope of instruction tuning.

Original: Ideally, we would have performed the Figure 6 ablation with cross-validation instead of a single split, which likely would have smoothed out that artifact.
Revised: Ideally, we would have performed the Figure 7 ablation with cross-validation instead of a single split, which likely would have smoothed out that artifact.

Original: G Q UALITATIVE E XAMPLES
Revised: F Q UALITATIVE E XAMPLES

Original: This section shows qualitative examples of FLAN responding to various prompts.
Revised: This section shows qualitative examples of FLAN responding to various prompts.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Revised: Generate utterances with the intent “where to vote in the state Senate

Original: [Figure]
Revised: [Figure]

Original: Generate utterances with the intent “where to vote in the state Senate election" ● Tell me where to vote in the state senate election ● Find out where to vote in the state senate election ● Where am i supposed to vote in the state senate election ● Where can i get information on where to vote in the state Senate election?
Revised: " ● Tell me where to vote in the state senate election ● Find out where to vote in the state senate election ● Where am i supposed to vote in the state senate election ● Where can i get information on where to vote in the state Senate election?

Original: Input:
Revised: Input:

Original: FLAN output:
Revised: FLAN output:

Original: Generate utterances with theintent “get COVID vaccine"
Revised: Generate utterances with the

Original: [Figure]
Revised: [Figure]

Original: ● I
Revised: “get COVID

Original: [Figure]
Revised: [Figure]

Original: like to get the covidvaccine.
Revised: to get the covid

Original: [Figure]
Revised: [Figure]

Original: I would likea COVID vaccine as soon as possible ● Can I get
Revised: like

Original: [Figure]
Revised: [Figure]

Original: COVID 
Revised: COVID vaccine

Original: [Figure]
Revised: [Figure]

Original: as

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: to

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Figure]

Revised: Yosemite

Original: [Figure]
Revised: [Figure]

Original: writing the story
Revised: writing the story,

Original: [Figure]
Revised: [Figure]

Original: FLAN output (cherrypicked):
Revised: output ( [Figure]

Original: [Figure]
Revised: [Figure]

Original: Write  sad story 
Revised: Write a sad story about carrots.

Original: The sun had barely risen when the carrots were taken from their beds and brought to market.
Revised: The sun had barely risen when the carrots were taken from their beds and brought to market.

Original: They'd had no time to gather their strength.
Revised: They'd had no time to gather their strength.

Original: They smelled of earth, their greens still wet from the dew.
Revised: They smelled of earth, their greens still wet from the dew.

Original: At ﬁrst no one was interested in buying them.
Revised: At ﬁrst no one was interested in buying them.

Original: But one man ﬁnally reached out and picked up one of the carrots.
Revised: But one man ﬁnally reached out and picked up one of the carrots.

Original: "I'll take this one!”
Revised: "I'll take this one!”

Original: he cried out.
Revised: he cried out.

Original: "It's for my mother!
Revised: "It's for my mother!

Original: I'll pay you well for it.”
Revised: I'll pay you well for it.”

Original: The man took the carrot to the cashier, eager to get it home to his mother.
Revised: The man took the carrot to the cashier, eager to get it home to his mother.

Original: He was careful to wrap it carefully so that it wouldn't get bruised or dirty.
Revised: He was careful to wrap it carefully so that it wouldn't get bruised or dirty.

Original: He hurried home, and when he saw his mother, he gave her the carrot.
Revised: He hurried home, and when he saw his mother, he gave her the carrot.

Original: “Is this one of the ones I grew?
Revised: “Is this one of the ones I grew?

Original: she asked, her voice wavering with emotion.
Revised: she asked, her voice wavering with emotion.

Original: "Yes, mother," he replied.
Revised: "Yes, mother," he replied.

Original: "It is, dear."
Revised: "It is, dear."

Original: The man left the carrot with his mother and hurried back to market.
Revised: The man left the carrot with his mother and hurried back to market.

Original: His mother was very sick, and he was afraid she wouldn't last much longer.
Revised: His mother was very sick, and he was afraid she wouldn't last much longer.

Original: He needed to buy all the carrots in market so that none would be left when she was gone.
Revised: He needed to buy all the carrots in market so that none would be left when she was gone.

Original: [Figure]
Revised: [Figure]

Original: John moved to mountain view.
Revised: John moved to mountain view.

Original: He spent friday night in San Francisco.
Revised: He spent friday night in San Francisco.

Original: He got a COVID vaccine.
Revised: He got a COVID vaccine.

Original: He voted in the state senate election.
Revised: He voted in the state senate election.

Original: Write a story that follows the above plot
Revised: Write a story that follows the above plot.

Original: Input:
Revised: Input:

Original: John arrived at the city with his vaccine, feeling prepared enough for the state senate election of San Francisco.
Revised: John arrived at the city with his vaccine, feeling prepared enough for the state senate election of San Francisco.

Original: He had voted for the ﬁrst time last night and was feeling good about it.
Revised: He had voted for the ﬁrst time last night and was feeling good about it.

Original: He felt more prepared than ever to win, and win big, because he was getting a COVID vaccine.
Revised: He felt more prepared than ever to win, and win big, because he was getting a COVID vaccine.

Original: He knew that his opponent had not gotten a COVID vaccine, and he knew he had already defeated him before the election began.
Revised: He knew that his opponent had not gotten a COVID vaccine, and he knew he had already defeated him before the election began.

Original: FLAN output (cherrypicked):
Revised: FLAN output (cherrypicked):

Revised: What is Barack Obama’s profession?

Revised: Answer in Danish

Original: [Figure]
Revised: [Figure]

Original: is
Revised: Input:

Original: [Figure]
Revised: [Figure]

Original: in Danish
Revised: output:

Original: [Figure]
Revised: [Figure]

Revised: What is

Revised: [Figure]

Revised: word in this sentence?

Revised: [Figure]

Revised: output:

Revised: C HANGES FROM V4 TO V

Revised: • Replaced the tables in the main ﬁgure with a ﬁgure, which takes up less space and focuses on zero-shot performance.

Revised: • Added GLaM 64B/64E as a baseline.

Revised: • Moved the ablation about the role of instructions, as well as prompt tuning, into the main paper (and condensed the ﬁgures).

Revised: C HANGES TO V4 FROM V

Revised: • We added a Frequently Asked Questions section (Appendix E).

Revised: • We added a section with qualitative examples (Appendix F).

Revised: • We added an additional ablation study on the role of instructions during ﬁnetuning (Appendix B.2).

Revised: • We updated the related work (Appendix D) with manuscripts posted on arxiv since our initial preprint.

Revised: C HANGES TO V3 FROM V

Revised: • The number of tokens used in pretraining was corrected from 2.81T to 2.49T tokens.

Revised: C HANGES TO V2 FROM V

Revised: • We updated the terminology to “datasets” and “task clusters.”

Revised: • We renamed the previous “open-domain QA” task cluster to “closed-book QA.”

Revised: • We extended the related work section and moved it to the Appendix D, using a shorter version in the main body.

Revised: • We added FLAN and LaMDA-PT results for additional datasets for which GPT-3 results were not reported • For TriviaQA, v1 reported results on the tfds dev set of 11,313 examples.

Revised: GPT-3 actually evaluates on the wikipedia dev set of 7,993 examples, so we ran an additional evaluation on that dev set in order to compare with GPT-3’s performance.

Revised: Zero-shot FLAN now beats zero-shot GPT-3 on that task (and therefore on 20 of 25 tasks).

Revised: We still show the original result in Table 2, though there is no GPT-3 result to compare with.

Revised: • We moved commonsense reasoning and coreference resolution from the main body to the Appendix.

Revised: • We moved prompt tuning from the main body to §4.5.

Revised: • We added data contamination analysis (Appendix C).

Revised: • We added few-shot instruction tuning (§4.4).

Revised: • We cited additional datasets in Appendix G. •

Revised: The number of tokens used in pretraining was corrected from 2.81T to 2.49T tokens.

Original: H T ASKS AND D ATASETS
Revised: G T ASKS AND D ATASETS

Original: This appendix further details the datasets that we use in this paper.
Revised: This appendix further details the datasets that we use in this paper.

Original: We group datasets into one of the following task clusters:
Revised: We group datasets into one of the following task clusters:

Original: • Natural language inference concerns how two sentences relate, typically asking, given a ﬁrst sentence, whether a second sentence is true, false, or possibly true.
Revised: • Natural language inference concerns how two sentences relate, typically asking, given a ﬁrst sentence, whether a second sentence is true, false, or possibly true.

Original: We use the following datasets: 1.
Revised: We use the following datasets: 1.

Original: ANLI (Nie et al., 2020) 2.
Revised: ANLI (Nie et al., 2020) 2.

Original: CB (De Marneffe et al., 2019) 3.
Revised: CB (De Marneffe et al., 2019) 3.

Original: MNLI (Williams et al., 2018) 4.
Revised: MNLI (Williams et al., 2018) 4.

Original: QNLI (Rajpurkar et al., 2018) 5.
Revised: QNLI (Rajpurkar et al., 2018) 5.

Original: SNLI (Bowman et al., 2015) 6.
Revised: SNLI (Bowman et al., 2015) 6.

Original: WNLI (Levesque et al., 2012) 7.
Revised: WNLI (Levesque et al., 2012) 7.

Original: RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009)
Revised: RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009)

Original: • Reading comprehension tests the ability to answer a question when given a passage that contains the answer.
Revised: • Reading comprehension tests the ability to answer a question when given a passage that contains the answer.

Original: We use the following datasets: 1.
Revised: We use the following datasets: 1.

Original: BoolQ Clark et al.
Revised: BoolQ Clark et al.

Original: (2019a) 2.
Revised: (2019a) 2.

Original: DROP (Dua et al., 2019) 3.
Revised: DROP (Dua et al., 2019) 3.

Original: MultiRC (Khashabi et al., 2018) 4.
Revised: MultiRC (Khashabi et al., 2018) 4.

Original: OBQA (Mihaylov et al., 2018) 5.
Revised: OBQA (Mihaylov et al., 2018) 5.

Original: SQuADv1 (Rajpurkar et al., 2016) 6.
Revised: SQuADv1 (Rajpurkar et al., 2016) 6.

Original: SQuADv2 (Rajpurkar et al., 2018)
Revised: SQuADv2 (Rajpurkar et al., 2018)

Original: • Commonsense reasoning evaluates the ability to perform physical or scientiﬁc reasoning with an element of common sense.
Revised: • Commonsense reasoning evaluates the ability to perform physical or scientiﬁc reasoning with an element of common sense.

Original: We use the following datasets: 1.
Revised: We use the following datasets: 1.

Original: COPA (Roemmele et al., 2011) 2.
Revised: COPA (Roemmele et al., 2011) 2.

Original: HellaSwag (Zellers et al., 2019) 3.
Revised: HellaSwag (Zellers et al., 2019) 3.

Original: PiQA (Bisk et al., 2020) 4.
Revised: PiQA (Bisk et al., 2020) 4.

Original: StoryCloze
Revised: StoryCloze

Original: (Mostafazadeh et al., 2016)
Revised: (Mostafazadeh et al., 2016)

Original: • Sentiment analysis is a classic NLP task aims to understand whether a piece of text is positive or negative.
Revised: • Sentiment analysis is a classic NLP task aims to understand whether a piece of text is positive or negative.

Original: We use the following datasets: 1.
Revised: We use the following datasets: 1.

Original: IMDB (Maas et al., 2011) 2.
Revised: IMDB (Maas et al., 2011) 2.

Original: Sentiment

Revised: Sentiment

Original: (Go et al., 2009) 3.
Revised: (Go et al., 2009) 3.

Original: SST-2 (Socher et al., 2013) 4.
Revised: SST-2 (Socher et al., 2013) 4.

Original: Yelp (Fast.
Revised: Yelp (Fast.

Original: AI) • Closed-book QA asks models to answer questions about the world without speciﬁc access to information that contains the answer.
Revised: AI) • Closed-book QA asks models to answer questions about the world without speciﬁc access to information that contains the answer.

Original: We use the following datasets: 1.
Revised: We use the following datasets: 1.

Original: ARC (Clark et al., 2018) 2.
Revised: ARC (Clark et al., 2018) 2.

Original: NQ (Lee et al., 2019; Kwiatkowski et al., 2019) 3. TriviaQA
Revised: NQ (Lee et al., 2019; Kwiatkowski et al., 2019) 3. TriviaQA

Original: Joshi et al.
Revised: Joshi et al.

Original: (2017) • Paraphrase detection asks a model to determine whether two sentences are semantically equivalent.
Revised: (2017) • Paraphrase detection asks a model to determine whether two sentences are semantically equivalent.

Original: 5 We use the following datasets: 1.
Revised: 4 We use the following datasets: 1.

Original: MRPC (Dolan & Brockett, 2005) 2.
Revised: MRPC (Dolan & Brockett, 2005) 2.

Original: QQP (Wang et al., 2018, see) 3.
Revised: QQP (Wang et al., 2018, see) 3.

Original: Paws Wiki (Zhang et al., 2019) • Coreference resolution tests the ability to identify expressions of the same entity in some given text.
Revised: Paws Wiki (Zhang et al., 2019) • Coreference resolution tests the ability to identify expressions of the same entity in some given text.

Original: We use the following datasets: 1.
Revised: We use the following datasets: 1.

Original: DPR (Rahman & Ng, 2012) 2.
Revised: DPR (Rahman & Ng, 2012) 2.

Original: Winogrande (Sakaguchi et al., 2020)
Revised: Winogrande (Sakaguchi et al., 2020)

Original: WSC273 (Levesque et al., 2012)
Revised: WSC273 (Levesque et al., 2012)

Original: • Reading comprehension with commonsense combines elements of both reading comprehension with commonsense.
Revised: • Reading comprehension with commonsense combines elements of both reading comprehension with commonsense.

Original: We use the following datasets: 1. CosmosQA
Revised: We use the following datasets: 1. CosmosQA

Original: (Huang et al., 2019) 2.
Revised: (Huang et al., 2019) 2.

Original: ReCoRD (Zhang et al., 2018)
Revised: ReCoRD (Zhang et al., 2018)

Original: • Struct to text tests the ability to describe some structured data using natural language.
Revised: • Struct to text tests the ability to describe some structured data using natural language.

Original: We use the following datasets: 1.
Revised: We use the following datasets: 1.

Original: CommonGen (Lin et al., 2020) 2.
Revised: CommonGen (Lin et al., 2020) 2.

Original: DART
Revised: DART

Original: (Nan et al., 2021) 3.
Revised: (Nan et al., 2021) 3.

Original: E2ENLG (Dušek et al., 2019) 4.
Revised: E2ENLG (Dušek et al., 2019) 4.

Original: WebNLG (Gardent et al., 2017) • Translation is the task of translating text from one language into a different language.
Revised: WebNLG (Gardent et al., 2017) • Translation is the task of translating text from one language into a different language.

Original: We use the following datasets: 1.
Revised: We use the following datasets: 1.

Original: En–Fr from WMT’14 (Bojar et al., 2014) 2.
Revised: En–Fr from WMT’14 (Bojar et al., 2014) 2.

Original: En–De, En–Tr, En–Cs, En–Fi, En–Ro, and En–Ru from WMT’16 (Bojar et al., 2016) 3.
Revised: En–De, En–Tr, En–Cs, En–Fi, En–Ro, and En–Ru from WMT’16 (Bojar et al., 2016) 3.

Original: En–Es from Paracrawl (Bañón et al., 2020)
Revised: En–Es from Paracrawl (Bañón et al., 2020)

Original: • Summarization asks models to read a piece of text and generate an abbreviated summary of it.
Revised: • Summarization asks models to read a piece of text and generate an abbreviated summary of it.

Original: We use the
Revised: We use the

Original: following
Revised: following

Original: datasets:
Revised: datasets:

Original: AESLC (Zhang & Tetreault, 2019) 2.
Revised: AESLC (Zhang & Tetreault, 2019) 2.

Original: CNN-DM (See et al., 2017) 3.
Revised: CNN-DM (See et al., 2017) 3.

Original: Gigaword (Napoles et al., 2012) 4.
Revised: Gigaword (Napoles et al., 2012) 4.

Original: MultiNews (Fabbri et al., 2019) 5.
Revised: MultiNews (Fabbri et al., 2019) 5.

Original: Newsroom (Grusky et al., 2018) 6.
Revised: Newsroom (Grusky et al., 2018) 6.

Original: Samsum (Gliwa et al., 2019) 7.
Revised: Samsum (Gliwa et al., 2019) 7.

Original: XSum (Narayan et al., 2018) 8.
Revised: XSum (Narayan et al., 2018) 8.

Original: AG News (Zhang et al., 2015) 9.
Revised: AG News (Zhang et al., 2015) 9.

Original: Opinion Abstracts - Rotten Tomatoes (Wang & Ling, 2016) 10.
Revised: Opinion Abstracts - Rotten Tomatoes (Wang & Ling, 2016) 10.

Original: Opinion Abstracts - iDebate (Wang & Ling, 2016) 11.
Revised: Opinion Abstracts - iDebate (Wang & Ling, 2016) 11.

Original: Wiki Lingua English (Ladhak et al., 2020) • Additional datasets that we assign to a miscellaneous task cluster include: 1.
Revised: Wiki Lingua English (Ladhak et al., 2020) • Additional datasets that we assign to a miscellaneous task cluster include: 1.

Original: Conversational question-answering:
Revised: Conversational question-answering:

Original: QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) 2.
Revised: QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) 2.

Original: Evaluating context-sentence word meanings: WiC (Pilehvar & Camacho-Collados, 2019) 3.
Revised: Evaluating context-sentence word meanings: WiC (Pilehvar & Camacho-Collados, 2019) 3.

Original: Question classiﬁcation: TREC (Li & Roth, 2002; Hovy et al., 2001) 4.
Revised: Question classiﬁcation: TREC (Li & Roth, 2002; Hovy et al., 2001) 4.

Original: Linguistic acceptability: CoLA (Warstadt et al., 2019) 5.
Revised: Linguistic acceptability: CoLA (Warstadt et al., 2019) 5.

Original: Math questions (Saxton et al., 2019)
Revised: Math questions (Saxton et al., 2019)

Original: For all tasks, our ﬁnetuning and evaluation code uses tensorﬂow datasets (TFDS) to load and process datasets.
Revised: For all tasks, our ﬁnetuning and evaluation code uses tensorﬂow datasets (TFDS) to load and process datasets.

Original: Regarding the number of training examples per dataset, we limited the training set size per dataset to 30,000 so that no dataset dominated the ﬁnetuning distribution.
Revised: Regarding the number of training examples per dataset, we limited the training set size per dataset to 30,000 so that no dataset dominated the ﬁnetuning distribution.

Original: When a test set with labels was available in TFDS, we used it; otherwise, we used the TFDS validation set as our test set, splitting the training set into a train and dev set.
Revised: When a test set with labels was available in TFDS, we used it; otherwise, we used the TFDS validation set as our test set, splitting the training set into a train and dev set.

Original: On the following pages, we show inputs and outputs for evaluation tasks where we compared with
Revised: On the following pages, we show inputs and outputs for evaluation tasks where we compared with

Original: GPT-3.
Revised: GPT-3.

Original: See the attached supplementary material for the templates for all other datasets.
Revised: See the attached supplementary material for the templates for all other datasets.

Original: H. N ATURAL L ANGUAGE I NFERENCE I NPUT
Revised: G. N ATURAL L ANGUAGE I NFERENCE I NPUT

Original: Joey Heindle (born 14 May 1993 in Munich) is a German singer.
Revised: Joey Heindle (born 14 May 1993 in Munich) is a German singer.

Original: He is best known for winning the seventh season of the game show Ich bin ein Star – Holt mich hier raus!
Revised: He is best known for winning the seventh season of the game show Ich bin ein Star – Holt mich hier raus!

Original: and ﬁnishing in 5th place in season 9 of Deutschland sucht den Superstar, despite universally negative reviews from the jury each week.
Revised: and ﬁnishing in 5th place in season 9 of Deutschland sucht den Superstar, despite universally negative reviews from the jury each week.

Original: Based on the paragraph above can we conclude that "Joey Heindle was highly disliked by people on television."?
Revised: Based on the paragraph above can we conclude that "Joey Heindle was highly disliked by people on television."?

Original: OPTIONS: -Yes -It’simpossible to say - No T ARGET Yes
Revised: OPTIONS: -Yes -It’simpossible to say - No T ARGET Yes

Original: I NPUT
Revised: I NPUT

Original: A: so I watch the ﬁsh, you know
Revised: A: so I watch the ﬁsh, you know.

Original: Whatever I can do to keep myself occupied
Revised: Whatever I can do to keep myself occupied.

Original: I like to have the
Revised: I like to have the

Original: TV on, because that usually keeps me, um, more occupied.
Revised: TV on, because that usually keeps me, um, more occupied.

Original: It kind of takes the time away and
Revised: It kind of takes the time away and

Original: I don’t realize , that’s really the only time I ever watch TV, is when I’m on the bike.
Revised: I don’t realize, that’s really the only time I ever watch TV, is when I’m on the bike.

Original: and then usually after I’m done riding the bike, just to cool myself down, I usually take a walk, you know, and that just kind of uh, gets me, you know, to where I’m not quite as tired I guess.
Revised: and then usually after I’m done riding the bike, just to cool myself down, I usually take a walk, you know, and that just kind of uh, gets me, you know, to where I’m not quite as tired I guess.

Original: But it’s deﬁnitely a task.
Revised: But it’s deﬁnitely a task.

Original: B: You think so?A:
Revised: B: You think so?

Original: I
Revised: A

Original:  can’t say that I really enjoy it.
Revised: : I can’t say that I really enjoy it.

Original: Based on the paragraph above can we conclude that "she really enjoys it"?
Revised: Based on the paragraph above can we conclude that "she really enjoys it"?

Original: OPTIONS:
Revised: OPTIONS:

Original: - Yes - No - It’s impossible to say
Revised: - Yes - No - It’s impossible to say

Original: T ARGET
Revised: T ARGET

Original: No
Revised: No

Original: I NPUT
Revised: I NPUT

Original: After years of study, the Vatican’s doctrinal congregation has sent church leaders a conﬁdential document concluding that "sex-change" procedures do not change a person’s gender in the eyes of the church.
Revised: After years of study, the Vatican’s doctrinal congregation has sent church leaders a conﬁdential document concluding that "sex-change" procedures do not change a person’s gender in the eyes of the church.

Original: Based on the paragraph above can we conclude that "Sex-change operations become more common."?
Revised: Based on the paragraph above can we conclude that "Sex-change operations become more common."?

Original: OPTIONS:
Revised: OPTIONS:

Original: - yes - no
Revised: - yes - no

Original: [Table]
Revised: [Table]

Original: H. R EADING C OMPREHENSION I NPUT
Revised: G. R EADING C OMPREHENSION I NPUT

Original: There are four ways an individual can acquire Canadian citizenship: by birth on Canadian soil; by descent (being born to a Canadian parent); by grant (naturalization); and by adoption.
Revised: There are four ways an individual can acquire Canadian citizenship: by birth on Canadian soil; by descent (being born to a Canadian parent); by grant (naturalization); and by adoption.

Original: Among them, only citizenship by birth is granted automatically with limited exceptions, while citizenship by descent or adoption is acquired automatically if the speciﬁed conditions have been met.
Revised: Among them, only citizenship by birth is granted automatically with limited exceptions, while citizenship by descent or adoption is acquired automatically if the speciﬁed conditions have been met.

Original: Citizenship by grant, on the other hand, must be approved by the Minister of Immigration,
Revised: Citizenship by grant, on the other hand, must be approved by the Minister of Immigration,

Original: Refugees and Citizenship.
Revised: Refugees and Citizenship.

Original: Can we conclude that can i get canadian citizenship if my grandfather was canadian?
Revised: Can we conclude that can i get canadian citizenship if my grandfather was canadian?

Original: OPTIONS: [Table]
Revised: [Table] [Table]

Original: I NPUT
Revised: I NPUT

Original: Imagine you are standing in a farm ﬁeld in central Illinois.
Revised: Imagine you are standing in a farm ﬁeld in central Illinois.

Original: The land is so ﬂat you can see for miles and miles.
Revised: The land is so ﬂat you can see for miles and miles.

Original: On a clear day, you might see a grain silo 20 miles away.
Revised: On a clear day, you might see a grain silo 20 miles away.

Original: You might think to yourself, it sure is ﬂat around here.
Revised: You might think to yourself, it sure is ﬂat around here.

Original: If you drive one hundred miles to the south, the landscape changes.
Revised: If you drive one hundred miles to the south, the landscape changes.

Original: In southern Illinois, there are rolling hills.
Revised: In southern Illinois, there are rolling hills.

Original: Why do you think this is?
Revised: Why do you think this is?

Original: What could have caused these features?
Revised: What could have caused these features?

Original: There are no big rivers that may have eroded and deposited this material.
Revised: There are no big rivers that may have eroded and deposited this material.

Original: The ground is capable of supporting grass and trees, so wind erosion would not explain it.
Revised: The ground is capable of supporting grass and trees, so wind erosion would not explain it.

Original: To answer the question, you need to go back 12,000 years.
Revised: To answer the question, you need to go back 12,000 years.

Original: Around 12,000 years ago, a giant ice sheet covered much of the Midwest United States.
Revised: Around 12,000 years ago, a giant ice sheet covered much of the Midwest United States.

Original: Springﬁeld, Illinois, was covered by over a mile of ice.
Revised: Springﬁeld, Illinois, was covered by over a mile of ice.

Original: Its hard to imagine a mile thick sheet of ice.
Revised: Its hard to imagine a mile thick sheet of ice.

Original: The massive ice sheet, called a glacier, caused the features on the land you see today.
Revised: The massive ice sheet, called a glacier, caused the features on the land you see today.

Original: Where did glaciers go?
Revised: Where did glaciers go?

Original: Where can you see them today?
Revised: Where can you see them today?

Original: Glaciers are masses of ﬂowing ice.
Revised: Glaciers are masses of ﬂowing ice.

Original: Question: "How big were the glaciers?"
Revised: Question: "How big were the glaciers?"

Original: Response: "One mile"
Revised: Response: "One mile"

Original: Does the response correctly answer the question?
Revised: Does the response correctly answer the question?

Original: [Table]
Revised: [Table]

Original: -no -yes

Original: T

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  soil is a renewable resource for growing plants
Revised: NPUT soil is a renewable resource for growing plants

Original: A plant that needs to expand will be able to have an endless resource in
Revised: A plant that needs to expand will be able to have an endless resource in

Original: OPTIONS:
Revised: OPTIONS:

Original: - dirt - pesticides - pay - beans [Table] ARGET 
Revised: - dirt - pesticides - pay - beans T ARGET dirt

Original: [Table]
Revised: G. C OMMONSENSE R EASONING

Original: [Table]
Revised: [Table]

Original: I packed up my belongings.
Revised: I packed up my belongings.

Original: What is the cause
Revised: What is the cause?

Original: [Table]
Revised: [Table]

Original: -Iwashunting for a new apartment.
Revised: -Iwashunting for a new apartment.

Original: - I was moving out of my apartment.
Revised: - I was moving out of my apartment.

Original: [Table] was
Revised: T ARGET

Original: [Table]
Revised: [Table]

Original:  out of my apartment.
Revised: moving out of my apartment.

Original: [Table]
Revised: [Table]

Original: What happens next in this paragraph?
Revised: What happens next in this paragraph?

Original: Once the rope is inside the hook, he begins moving up the wall but shortly after he stops and begins talking.
Revised: Once the rope is inside the hook, he begins moving up the wall but shortly after he stops and begins talking.

Original: The male then begins talking about the clip again and goes back up the wall.
Revised: The male then begins talking about the clip again and goes back up the wall.

Original: as he
Revised: as he

Original: [Table]
Revised: [Table]

Original: - progresses, there are hooks everywhere on the wall and when he gets near them, he puts his rope inside of it for support and safety. - changes time, an instant replay of his initial move is shown a second time.
Revised: - progresses, there are hooks everywhere on the wall and when he gets near them, he puts his rope inside of it for support and safety. - changes time, an instant replay of his initial move is shown a second time.

Original: - continues to talk, another male speaks about the move and shows another closeup of the plex by the male. - continues, other people start to arrive and begin to hang out with him as he makes a few parts of the rope.
Revised: - continues to talk, another male speaks about the move and shows another closeup of the plex by the male. - continues, other people start to arrive and begin to hang out with him as he makes a few parts of the rope.

Original: [Table]
Revised: [Table]

Original: progresses, there are hooks everywhere on the wall and when he gets near them, he puts his rope inside of it for support and safety.
Revised: progresses, there are hooks everywhere on the wall and when he gets near them, he puts his rope inside of it for support and safety.

Original: [Table]
Revised: [Table]

Original: Here is a goal: Remove smell from garbage disposal.
Revised: Here is a goal: Remove smell from garbage disposal.

Original: How would you accomplish this goal
Revised: How would you accomplish this goal?

Original: [Table]
Revised: [Table]

Original: -Createsoda ice cubes and grind through disposal. - Create vinegar ice cubes and grind through disposal.
Revised: -Createsoda ice cubes and grind through disposal. - Create vinegar ice cubes and grind through disposal.

Original: [Table]
Revised: [Table]

Original: ARGET

Original: Create vinegar ice cubes and grind through disposal.
Revised: Create vinegar ice cubes and grind through disposal.

Original: [Table]
Revised: [Table]

Original: Caroline never drinks carbonated beverages. 
Revised: Caroline never drinks carbonated beverages. Her

Original: Her friends pick on her because ofit.
Revised: friends pick on her because of

Revised: it.

Original: One day they challenged her to drink a soda.
Revised: One day they challenged her to drink a soda.

Original: Caroline wanted to win the challenge.
Revised: Caroline wanted to win the challenge

Original: Predict the next sentence.
Revised: Predict the next sentence.

Original: [Table]
Revised: [Table]

Original: -Carolinerefused to open the soda.
Revised: -Carolinerefused to open the soda.

Original: - Caroline opened the soda and drank it all in one gulp!
Revised: - Caroline opened the soda and drank it all in one gulp!

Original: [Table]
Revised: [Table]

Original: Caroline opened the soda and drank it all in one gulp!
Revised: Caroline opened the soda and drank it all in one gulp!

Original: [Table]
Revised: [Table]

Original: What season is the Northern Hemisphere experiencing when it is tilted directly toward the Sun?
Revised: What season is the Northern Hemisphere experiencing when it is tilted directly toward the Sun?

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Question: who is the girl in more than you know??
Revised: Question: who is the girl in more than you know??

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Please answer this question: Henry Croft, an orphan street sweeper who collected money for charity, is associated with what organised charitable tradition of working class culture in London,
Revised: Please answer this question: Henry Croft, an orphan street sweeper who collected money for charity, is associated with what organised charitable tradition of working class culture in London,

Original: England?
Revised: England?

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  C OREFERENCE R ESOLUTION
Revised: G. C OREFERENCE R ESOLUTION

Original: [Table]
Revised: [Table]

Original: How does the sentence end
Revised: How does the sentence end?

Original: [Table]

Original: Elena wanted to move out of her parents fast but Victoria wanted to stay for a while,
Revised: Elena wanted to move out of her parents fast but Victoria wanted to stay for a while,

Original: [Table]
Revised: [Table]

Original: -Elenawent to school.
Revised: -Elenawent to school.

Original: - Victoria went to school.
Revised: - Victoria went to school.

Original: [Table]
Revised: [Table]

Original: Victoria went to school.
Revised: Victoria went to school.

Original: Winogrande (
Revised: Winogrande (

Original: [Table]
Revised: [Table]

Original: Jane knocked on Susan’s door, but there was no answer.
Revised: Jane knocked on Susan’s door, but there was no answer.

Original: [Table]
Revised: [Table]

Original: - Jane was out.
Revised: - Jane was out.

Original: - Susan was out
Revised: - Susan was out.

Original: [Table]
Revised: [Table]

Revised: Susan was out.

Original: H. R EADING C OMPREHENSION WITH C OMMONSENSE I NPUT
Revised: G. R EADING C OMPREHENSION WITH C OMMONSENSE I NPUT

Original: Complete the passage.
Revised: Complete the passage.

Original: (CNN) –
Revised: (CNN) –

Original: At ﬁrst glance, "The Flat" might seem like an episode of "Hoarders," Israeli-style.
Revised: At ﬁrst glance, "The Flat" might seem like an episode of "Hoarders," Israeli-style.

Original: The documentary ﬁlm opens after an elderly woman dies in Tel Aviv.
Revised: The documentary ﬁlm opens after an elderly woman dies in Tel Aviv.

Original: Her grandchildren assemble to clean out her apartment, packed with dusty books, vintage clothing (dozens of pairs of fancy gloves, for instance), enough purses to stock a department store, jewelry, mementoes and closets full of knickknacks.
Revised: Her grandchildren assemble to clean out her apartment, packed with dusty books, vintage clothing (dozens of pairs of fancy gloves, for instance), enough purses to stock a department store, jewelry, mementoes and closets full of knickknacks.

Original: But buried among the detritus they chance upon something remarkable – mysterious papers linking the grandparents to an important Nazi ﬁgure.
Revised: But buried among the detritus they chance upon something remarkable – mysterious papers linking the grandparents to an important Nazi ﬁgure.

Original: How could such ardent
Revised: How could such ardent

Original: Zionists, who left their native Germany in the early 1930s, have been involved with an SS ofﬁcial like Leopold von Mildenstein?
Revised: Zionists, who left their native Germany in the early 1930s, have been involved with an SS ofﬁcial like Leopold von Mildenstein?

Original: What I found out was this journey, the Nazi (
Revised: What I found out was this journey, the Nazi (

Original: OPTIONS:
Revised: OPTIONS:

Original: - Arnon Goldﬁnger)
Revised: - Arnon Goldﬁnger)

Original: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.
Revised: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.

Original: - CNN)
Revised: - CNN)

Original: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.
Revised: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.

Original: - Germany)
Revised: - Germany)

Original: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.
Revised: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.

Original: - Israeli)
Revised: - Israeli)

Original: and his wife were accompanied by my grandparents," Goldﬁnger told CNN. -
Revised: and his wife were accompanied by my grandparents," Goldﬁnger told CNN. -

Original: Leopold von Mildenstein) and his wife were accompanied by my grandparents
Revised: Leopold von Mildenstein) and his wife were accompanied by my grandparents

Original: ," Goldﬁnger told CNN.
Revised: ," Goldﬁnger told CNN.

Original: - Nazi)
Revised: - Nazi)

Original: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.
Revised: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.

Original: - SS)
Revised: - SS)

Original: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.
Revised: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.

Original: - Tel Aviv)
Revised: - Tel Aviv)

Original: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.
Revised: and his wife were accompanied by my grandparents," Goldﬁnger told CNN.

Original: - The Flat) and his wife were accompanied by my grandparents," Goldﬁnger told CNN.
Revised: - The Flat) and his wife were accompanied by my grandparents," Goldﬁnger told CNN.

Original: - Zionists) and his wife were accompanied by my grandparents
Revised: - Zionists) and his wife were accompanied by my grandparents

Original: ," Goldﬁnger told CNN.
Revised: ," Goldﬁnger told CNN.

Original: T ARGET
Revised: T ARGET

Original: Leopold von Mildenstein) and his wife were accompanied by my grandparents
Revised: Leopold von Mildenstein) and his wife were accompanied by my grandparents," Goldﬁnger told

Original: Goldﬁnger told CNN.
Revised: CNN

Original: (ReCoRD).
Revised: (ReCoRD).

Original: ReCoRD (Zhang et al., 2018) asks for the answer to a cloze-style question where an entity is masked out.
Revised: ReCoRD (Zhang et al., 2018) asks for the answer to a cloze-style question where an entity is masked out.

Original: Of the the training set of 100,730 examples, we use 30,000 for train and 200 for dev.
Revised: Of the the training set of 100,730 examples, we use 30,000 for train and 200 for dev.

Original: We use the TFDS validation set of 10,000 examples as our test set for reporting numbers.
Revised: We use the TFDS validation set of 10,000 examples as our test set for reporting numbers.

Original: H. T RANSLATION (7 LANGUAGES ) I NPUT
Revised: G. T RANSLATION (7 LANGUAGES ) I NPUT

Original: Here the largest town of the district is located: Nordenham , lying opposite to Bremerhaven at the Weser mouth.
Revised: Here the largest town of the district is located: Nordenham , lying opposite to Bremerhaven at the Weser mouth.

Original: Translate to German T ARGET
Revised: Translate to German T ARGET

Original: An der B 211 beﬁndet sich in Loyermoor der so genannte “Geest-Abbruch”, der eine Höhendifferenz von gut 30 Meter überbrückt.
Revised: An der B 211 beﬁndet sich in Loyermoor der so genannte “Geest-Abbruch”, der eine Höhendifferenz von gut 30 Meter überbrückt.