  Can we enable NLP models to appropriately respond to instructional prompts
and consequently generalize to new tasks? To study this question, we leverage
the existing NLP datasets and the instructions that were used to crowdsource
them to create NATURAL INSTRUCTIONS, a dataset of instructions and
task-specific input/output data. This dataset consists of 61 distinct language
instructions and about 600k task instances, and is used to evaluate existing
state-of-the-art language-models (LMs) in addressing new tasks by few-shot
prompting of GPT3 and fine-tuning BART. Our analysis indicates that: (a) the
existing models indeed benefit from instructions and hence, show improved
generalization to new tasks; (b) while models like GPT-3 generally benefit from
instructions, the extent of their gains varies across different fields of
instructions and also depends on the task being solved; (c) generalization to
unseen tasks in NATURAL INSTRUCTIONS remains far from perfect for the
state-of-the-art, indicating significant room for more progress in this
direction.
