Learning from Task Descriptions (2020)

Abstract: Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. We instantiate this frame- work with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model’s ability to solve each task. Moreover, the dataset’s structure tests specific types of systematic generalization. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers.

## introduction the dominant paradigm in supervised nlp today is learning from examples, where machine learning algorithms are trained using a large set of taskspecific input-output pairs. in contrast, humans learn to perform the same task by reading a description, after which they are able to perform the task in a zero-shot manner-indeed, this is how crowd-sourced nlp datasets are constructed. in this paper, we argue that learning from task descriptions in this way is a necessary attribute of a general purpose nlp system, and we propose it as a new paradigm to train and test nlp systems. recent work in nlp has shown significant progress in learning tasks from examples. large pretrained language models have dramatically improved performance on standard benchmarks @xcite @xcite , 2019) and have shown promising results in zero shot prediction by leveraging their language understanding capabilities @xcite @xcite . despite this progress, there are many serious issues that come with learning from examples. there is an almost infinite number of tasks that a person might wish to solve with a general-purpose nlp system. learning to solve these tasks by reading a description instead of observing a collection of examples would solve the problem of having to create training sets for each language task. such a system would also be more accessible to practitioners and domain experts in other fields, who could describe their tasks and solve them, opening up new avenues of research where it is expensive or infeasible to gather training data. additionally, we find that current supervised learning techniques partly achieve their success due to memorizing uninteresting aspects of the training distribution @xcite @xcite . teaching a system to learn a task from the description alone would alleviate these biases, as new training data would not be needed to learn a novel task. in this paper, we synthesize prior approaches to zero-shot learning in nlp and provide a formal framework for thinking about the zero-shot prediction problem. we show that previous zero-shot approaches are limited in both scope of application and rigour of evaluation. for example, while prior work has used zero-shot prediction for text classification, entity typing, and relation extraction, we push this to the more complex task of slot filling. we instantiate our formalism in an english language dataset, zest (zero shot learning from task descriptions), that is formatted similarly to reading comprehension datasets, in that we formulate task descriptions as questions and pair them with paragraphs of text. we choose this format as it provides a natural way to crowdsource data. this zero-shot dataset differs from typical reading comprehension datasets, however, in that each task description is paired with twenty different passages, and we evaluate a model's ability to solve the task, not just give the correct answer for a single (question, passage) pair. that is, given a question, a model produces some decision function f , and it is this function which we comprehensively evaluate on many different inputs. we also carefully select axes on which to evaluate the generalization of a model to different kinds of task descriptions, changing task descriptions in specific ways to systematically push the field towards more interesting and complex task descriptions. we evaluate models based on recent state-ofthe-art sequence to sequence architectures, which seem most suited to the task of zero shot prediction in this setting. we find that our best model based on t5 @xcite achieves a score of only 12% on this data, leaving a significant gap to our human performance estimate of 42%. zero shot learning from complex task descriptions remains a significant challenge for current nlp systems. ## learning from task descriptions this section describes our framework for enabling zero-shot generalization to unseen tasks, and relates it to prior work. ## instantiating the framework section 2 showed a framework for training and testing a general purpose system that could perform unseen nlp tasks. an ideal system in this framework would be able to read the descriptions of the tasks in the glue suite @xcite and perform well with no additional training. however, this goal is far beyond the current capabilities of today's models. in order to make progress, we must break down the problem into manageable steps. in this section we outline the scope that we envision for a reasonable nlpfocused dataset that can push forward the current state of learning from task descriptions, without being so challenging as to be out of reach. sec. 4 describes the data collection process for zest, our new english benchmark built following this scope. to define the scope, we begin by considering the types of applications a model that could successfully learn from task descriptions might enable. the largest bottleneck in building nlp applications today is collecting labeled data. our framework would eliminate this step, making it possible to build ad hoc nlp applications to easily filter, categorize, or extract structured information from corpora. for example, when planning a camping trip, one might want to know "what are the names of all the campgrounds and their locations?" that are listed in a collection of documents, which specifies an ad hoc request to return all examples of the located at relationship between the campground and location entity types. accordingly, it's important to include examples of the basic task building blocks of such a system: classification, typed entity extraction, and relation extraction in a benchmark dataset. in doing so, it would unify the prior work in zero-shot nlp (sec. 2.4) that has focused on just a single task, and require a single model to be able to handle any of these tasks at test time, instead of separate models for each task. more concretely, as each task τ defines a set of allowable outputs @xmath0, we can mix multiple output sets y in a single dataset as long as the output set is specified in the task description. zest includes the most common output sets: discrete classes, lists of (optionally) typed spans from the input, and relationships between spans. examples of each are shown in to move beyond aggregating existing approaches into a single dataset, recall that in our framework observing the task description d τ in addition to the input x allows us to test a model's generalization relative to four variables: x, y, τ , and d τ (fig. 1 ). motivated by this observation, we propose an approach that systematically varies the task descriptions and inputs while controlling for other sources of variability in order to test whether a system can generalize in multiple ways. to implement this idea, we begin by collecting a set of task descriptions, d τ , inputs x, and associated outputs, y. this base group of instances already allows us to test performance of unseen tasks on un- generalization question input passage (shortened) answer base can i hike to a waterfall at this national park? ... yet here at whiskeytown nra, we encourage you to chase waterfalls -go visit them! whiskeytown has four major waterfalls ... yes paraphrase is there a waterfall to hike to at this national park? (same as above) yes semantic flips can i hike to a canyon at this national park? ... descending 1,300 feet (396 m) past a large alcove, the trail meanders in a wide canyon ... yes composition what time of year is best to see the popular waterfalls in this national park? ... two viewing platforms provide the best view of great falls. this overlook is the last place that the falls can be viewed ... na output structure what waterfall hikes are there in this national park and are they wheelchair accessible? ... bridalveil fall is often the first waterfall you'll see when entering ... although paved, this is trail is not wheelchair accessible due to its grade. paraphrase we can test generalization to changes in the task description d τ while keeping the task τ fixed by paraphrasing the description. by also fixing x, we can use these paraphrases to test whether a model consistently predicts the correct output given the same input and underlying task. as we collect applicable inputs x for a task using a retrieval mechanism given the task description (section 4), this also adds some lexical distance between the input and the description, to avoid simple lexical shortcuts to solving the task @xcite . ## collecting zest to illustrate our novel way of evaluating and framing the "learning from task descriptions" problem, we provide an empirical demonstration of where current systems fail by collecting a challenge dataset. we hope this will serve as a starting point for making progress towards this goal of learning from descriptions. in this section we describe our annotation efforts, which consist of our design for the dataset, as well as three crowdsourcing steps: collecting tasks (in question form), gathering relevant documents, and annotating answers for the (task, document) pairs. ## establishing a baseline this section describes our baseline model results. ## conclusion we introduced a framework for creating general purpose nlp systems that can solve tasks from natural language descriptions, synthesizing and extending previous work in zero-shot learning. to make progress toward this goal, we create a dataset, zest, that rigorously evaluates how well a model truly understands each task. the dataset is designed to test models' ability to systematically generalize across four different areas. stateof-the-art performance on zest is 12%, leaving much room for future improvement. while we have been focused on zero shot learning from task descriptions, our framework also permits few-shot scenarios where a task description is given along with a handful of examples, making meta-learning approaches applicable. this is an interesting avenue for future work, for which zest should also be useful. to facilitate future work, we make our models, code, and data available at https://allenai.org/data/ zest .