Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2022)

Abstract: Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.

## introduction we have witnessed great progress in solving many nlp datasets through fine-tuning pre-trained language models (lms) @xcite . more recent studies show tremendous promise in generalization within the set of observed tasks through multi-task training and unified encoding @xcite ., grammar check tagging essential phrases question typing answering questions i nput: she chose to make a salad for lunch on sunday. question: how long did it take for her to make a salad? crowdsourcing i nstr uction: list all the words that are essential for answering it correctly. [...] crowdsourcing i nstr uction: label the type of the temporal phenomena in the question. example are [...] output: 30mins output: making salad output: no ? supervision with seen tasks output: event duration ? evaluation on unseen tasks crowdsourcing i nstr uction: label "yes" if the sentence contains any grammatical issues. otherwise, [...] crowdsourcing i nstr uction: answer the provided question based on a given [...] figure 1 : we construct the natural instructions dataset from crowdsourcing instructions and instances of different nlp datasets. we study if models can learn from seen tasks and generalize to unseen tasks given their natural crowdsourcing instructions. 2021). however, cross-task generalization -generalization to unseen tasks -has generally remained under-explored. for example, can we supervise a model with instances of grammar checking or question answering tasks, yet expect it to solve a different task like question typing (fig. 1 ). evidently, humans are capable of such generalizations; an average human can follow natural language instructions to solve a variety of problems, as evident by the success of crowdsourcing platforms (also argued in @xcite ). in this paper, we study if models can generalize to unseen tasks given their crowdsourcing instructions (fig. 1 ). we build natural instructions, a dataset consisting of natural crowdsourcing instructions for various tasks and their instances. training on seen tasks t seen in our dataset, we build a model that learns to follow natural instructions that define a task and perform tasks (i.e., mapping input to output). testing on unseen tasks t unseen , we evaluate if the model can perform unseen tasks solely from task instance-level generalization task-level generalization training data x train , y train pit, x train t , y train t q t p tseen evaluation x ñ y where: px, yq p px test , y test q px, itq ñ y where: px, yq p px test t , y test t q t p tunseen @xcite a comparison of task vs instance-level generalization it, xt and yt indicate natural language instructions, input, and output sets respectively for task t. in the conventional setup, training and evaluation are done on the instances of the same task. however, in task-level generalization, a model is expected to generalize to unseen tasks, where tunseen x tseen" h. their instructions and without any task-specific labeled data ( we compile natural instructions from task instructions written by researchers for crowdsourcing existing nlp datasets. such crowdsourcing instructions often elaborate a variety of details about how a task should (and should not) be done. to provide a systematic study of various elements of crowdsourcing instructions, we map them to a unified schema to cover the most important elements of task descriptions -such as definition, constraints, positive and negative examples. we collect tasks in natural instructions as minimal stand-alone steps provided to crowdworkers to complete a downstream nlp task. for example, tasks collected from qasc @xcite include sub-tasks about generating topic words or combining facts, as well as answering multi-hop questions. therefore our dataset not only contains typical downstream tasks in nlp, but also the intermediate subtasks that are not well-represented in the common benchmarks. the unified schema and the collection of minimal subtasks enable training lms that can generalize across different tasks by learning from instructions. in total, our dataset consists of 61 distinct nlp tasks and 193k instances. our experimental results indicate that lms learn to leverage natural language instructions as they show improved generalization to new tasks. for example, a bart @xcite achieves a 19% gain in terms of cross-task generalization compared to a model not using instructions ( §6). importantly, lms can generalize better to unseen tasks if they observe more tasks in training (fig. 2b ). this upward trajectory suggests the potential for stronger cross-task generalizable models upon scaling up the diversity of tasks represented in a metadataset of task instructions. despite the benefits of instructions, we observe a sizable gap between models' generalization and their estimated upperbounds (6.4), encouraging the community to work on this challenging problem. contributions: in summary, the contributions of this work are as follows: (a) we introduce natural instructions, a dataset of humanauthored instructions curated from existing wellknown datasets mapped to a unified schema, providing training and evaluation data for learning from instructions; (b) we build models that can encode instructions and show: (b.1) the benefit of crosstask generalization by leveraging instructions; (b.2) the importance of different elements of instructions in the performance; (b.3) noteworthy headroom for improvement on our benchmark, which hopefully will motivate further work in this direction. ## related works learning from instructions. there is recent literature on the extent to which models follow language instructions @xcite @xcite . for example, efrat and levy (2020) examine if language models can follow crowdsourcing instructions with no further training. on the contrary, our work is pursuing a fundamentally different goal: creating a dataset of crowdsourcing instructions and task instances and formulating cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. @xcite construct a crowdsourced dataset with short question-like task descriptions. compared to this work, our instructions are longer, more complex and natural since they were used to collect datasets through crowdsourcing. promptsource and flan @xcite are two concurrent works that pursue a similar goal as ours. a key difference between our work to these works is in terms of data collection strategy. our work uses natural instructions created by nlp researchers before the dataset instances were created by crowd workers, and hence it contains the complete definition of each task (definition, things to avoid, negative examples, etc.). on the other hand, instructions in the concurrent work are collected retroactively based on the alreadyavailable task instances. our natural instructions enable evaluating models on how they learn tasks given different elements of task descriptions. (see §a.5 for further comparisons.) nevertheless, we believe that all these approaches to constructing instructions and task categories are complementary and the community will benefit from considering both towards solving the challenging problem of cross-task generalization. prompt engineering. constructing effective discrete prompts for language models to perform nlp tasks is an active area of research @xcite @xcite . such prompts are often extremely short and may not include a complete definition of complex tasks. in contrast, our instructions encode detailed instructions as they were used to collect the datasets. moreover, the goals are different: most prompt-engineering approaches seek prompts with higher performance on a particular task, typically through assumptions about their target task which make them non-trivial to generalize to any other task. however, our introduced meta dataset enables the measurement of generalization to unseen tasks. beyond standard multi-task learning. multitask learning is a long-standing goal for ai @xcite and has led to successful models that can support a wider range of tasks @xcite @xcite @xcite . most of the conventional setups in the multi-tasking literature evaluate on instances that belong to the tasks that are seen, i.e., their labeled instances were observed during training (1st column of ## defining cross-task generalization here we formally define the problem setup for generalization across tasks. each task t consists of input/output instances px t , y t q and is described in terms of its natural language instructions i t . task-specific models. standard supervised learning algorithms use task-specific labeled instances to learn a mapping from input x to output y: m pxq " y for px, yq p px train t , y train t q and is evaluated on the test instances of the same (or similar) task px test t , y test t q. we refer to this as the instance-level generalization ( cross-task models. in this setup, the goal is to learn a model m that at inference obtains the output y given the input x and the task instruction i t : m pi t , xq " y, for px, yq p px t , y t q. in contrast to the task-specific models, no task-specific training data is used to learn the mapping m . we collect natural instructions ( §4) to study this question: can a model be trained to follow instructions via training tasks t seen and be generalized to follow instructions for a task t 1 p t unseen . we refer to this as a task-level generalization ( ## natural instructions natural instructions consists of instructions that describe a task (e.g., question answering) and instances of that task (e.g., answers extracted for a given question). fig. 3 shows an example instruction for the task of 'generating questions that require an understanding of event duration' accompanied with positive and negative examples that contextualize the task. here we introduce a schema for representing instructions ( §4.1) and then describe how existing datasets (their crowdsourcing templates) are mapped into our schema ( §4.2). ## problem setup and models here we define different cross-task generalization settings ( §5.1) and the models ( §5.2). ## conclusion in this paper, we studied the goal of building models that generalize to new tasks by encoding and understanding crowdsourcing instructions. we introduced natural instructions, which is built based on existing crowdsourced datasets, that enables building such models and systematically evaluate them. to the best of our knowledge, this is the first work to show the benefit of instructions towards improved cross-task generalization. additionally, we observe that our proposed task has a large room for improvement, which we believe will bring more attention to building stronger models that can generalize to a wider range of tasks.