Super- N atural I nstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022)

Abstract: How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions—training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9% on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.

## introduction the nlp community has witnessed great progress in building models for generalization to unseen tasks via in-context instructions @xcite • input: "context: … 'that's fantastic, i'm glad we came to something we both agree with.' utterance: 'me too. i hope you have a wonderful camping trip.'" • output: "yes" • explanation: "the participant engages in small talk when wishing their opponent to have a wonderful trip." definition "... given an utterance and recent dialogue context containing past 3 utterances (wherever available), output 'yes' if the utterance contains the small-talk strategy, otherwise output 'no'. small-talk is a cooperative negotiation strategy. it is used for discussing topics apart from the negotiation, to build a rapport with the opponent." task instruction • input: "context: … 'i am excited to spend time with everyone from camp!' utterance: 'that's awesome! i really love being out here with my son. do you think you could spare some food?' " • expected output: "yes" positive examples negative examples ## related work language instructions are a versatile way of defining goals, which is why they have been studied in the context of a variety of applications, such as instructions in grounded environments @xcite @xcite and database commands @xcite . here, we focus on applications of instructions for general nlp tasks. recent literature has been motivated by building models that are generalizable across a variety of nlp tasks, when prompted with either a few examples @xcite or language definitions @xcite @xcite @xcite . our work is related to the existing benchmarks in this line of work, as delineated in concurrent to our work, promptsource @xcite is another benchmark of tasks and their language instructions (prompts). an important distinction between this benchmark and ours is the phrasing of the task definitions: while promptsource task definitions are relatively concise, our task definitions are collected with the intention of providing a complete definition of each task and therefore are longer (24 tokens vs. 56 tokens on average; beyond cross-task generalization, our benchmark can also be used to study multi-task learning more broadly, which is a longstanding goal for ai @xcite . traditionally, this literature focuses on setups that involve evaluation on tasks that are observed during training @xcite . more recent studies show promise that large-scale multi-task learning can enable strong generalization to similar tasks via unified encoding @xcite or better finetuning results on downstream tasks @xcite . our proposed benchmark provides diverse tasks for studying multi-tasking at a massive scale. ## super-naturalinstructions super-naturalinstructions is a metadataset @xcite consisting of a variety of nlp tasks (see fig. 2a ) and instructions that describe them in plain language. instruction schema. all task instructions follow the same uniform schema (see fig. 1 ) which is composed of the following parts: • definition defines a given task in natural language. this is a complete definition of how an input text (e.g., a sentence or a document) is expected to be mapped to an output text. ## tk-instruct: learning to follow instructions at scale defining generalization to unseen tasks. each task t is defined via its natural language instruction i t , and each task has a set of input/output instances (x t , y t ). a model m is expected to produce the output y, given the input x and the task instruction i t : m (i t , x) = y, for (x, y) ∈ (x t , y t ). in particular, we would like to evaluate model m on tasks that are not observed (i.e., their instances were not used for training m ). the only source of signal for learning the task at inference time is in-context instructions i t that contain a definition and demonstration examples of the task. ## further analysis we conduct further analysis to understand the important factors for models to generalize across tasks. due to the computational cost, this analysis is done on the english track and using the t5-3b checkpoint, except for the experiments on model sizes. ## conclusion we construct a large-scale benchmark consisting of a diverse set of nlp tasks and their instructions. this benchmark can serve as a rich playground for training or evaluation of models that can generalize to unseen tasks by following instructions. furthermore, we train tk-instruct using this data, and demonstrate its capability to perform unseen tasks to a surprising extent. we provide extensive analysis to understand the important factors for such generalization. we hope our data and model will facilitate future work towards more general-purpose models.