P rompt S ource: An Integrated Development Environment and Repository for Natural Language Prompts (2022)

Abstract: PromptSource is a system for creating, sharing, and using natural language prompts. Prompts are functions that map an example from a dataset to a natural language input and target output. Using prompts to train and query language models is an emerging area in NLP that requires new tools that let users develop and refine these prompts collaboratively. PromptSource addresses the emergent challenges in this new setting with (1) a templating language for defining data-linked prompts, (2) an interface that lets users quickly iterate on prompt development by observing outputs of their prompts on many examples, and (3) a community-driven set of guidelines for contributing new prompts to a common pool. Over 2,000 prompts for roughly 170 datasets are already available in PromptSource. PromptSource is available at https://github.com/bigscience-workshop/promptsource .

## introduction prompt engineering is emerging as a new focus in nlp, particularly in zero-and few-shot learning settings. prompting is the practice of representing a task as a natural language utterance in order to query a language model for a response @xcite . for example, if a language model is conditioned on the text "she hit a home run. the previous sentence is about ...", then the model's subsequent generation would be interpreted as a prediction of the topic of the preceding sentence, e.g. by mapping a response such as "sports" to a class label. in specific contexts, prompting has been shown to have advantages over traditional classification, for example facilitating adaptation of language models to ad-hoc tasks and improving sample efficiency in low-data settings @xcite @xcite . these advantages motivate a practical challenge: how can we enable users to create, refine, and share prompts? the process of prompt engineering is critical for successful deployment as choices in prompting can affect downstream predictions significantly, particularly in the zero-shot setting @xcite @xcite . furthermore, training directly on collections of prompts can enable large models to generalize to new prompts more robustly @xcite @xcite . there is therefore a growing need for tools that support the creation of corpora of prompts. promptsource is an integrated development environment and repository for natural language prompts to use in the context of zero-shot (or gradient-based few-shot) learning. it provides a web-based gui that enables developers to write prompts in a templating language and immediately view their outputs on different examples. the system is integrated with the huggingface datasets library @xcite , so that users can load any dataset automatically, browse existing prompts, and create new ones. through the course of writing thousands of prompts, we converged on three key aspects to the design of promptsource: prompt templates can easily be distributed and used in other systems. • tools for prompt management. prompt-source has multiple view to address the needs of prompt authors at different stages of the prompt engineering cycle. a global view lets authors browse datasets and existing prompt templates. a local view facilitates iteration on prompt wording and metadata, as well as testing on individual examples. promptsource includes a set of guidelines for prompting based on a large-scale prompt writing pilot. promptsource's collection is meant to be useful for a wide range of research, based on iterative refinement of a set of quality standards. prompts in promptsource are also annotated with various pieces of metadata to make finding and using prompts easier. the promptsource system includes over 2,000 open-source prompts for roughly 170 datasets, which have all been reviewed to meet the quality standards. this collection, which we call the public pool of prompts (p3), allows users to materialize prompted forms of datasets for hundreds of different tasks. the t0 series of models @xcite for zero-shot inference were fine-tuned on a subset of p3. since then, promptsource and p3 have been extended for research on multi-lingual prompting @xcite and priming, i.e., incontext few-shot learning @xcite . the promptsource system and associated content is a first step in the study of systems for prompt engineering, an area that is likely to continue to grow. ## background and related work promptsource builds on recent work in prompting and prompt engineering. it is also related to work on systems for other types of annotations. prompting recently, prompting has emerged as a new focus within nlp as it can dramatically improve language models' few-shot and zeroshot performance in a wide range of downstream tasks @xcite @xcite . prompts and prompt engineering come in several varieties @xcite . promptsource is focused on facilitating research with human-written prompts, in which natural language is the medium for describing tasks. this approach has the advantage that prompts can be understood, modified, and applied without being tied to a specific model. in contrast, past work has also aimed to automatically construct prompts by framing the search for a good prompt as a learning problem. these prompts can either be expressed in natural language @xcite or as arbitrary vectors (a.k.a. "continuous" or "soft" prompts) not corresponding to words in the model's original vocabulary @xcite when using human-written prompts, there are several possible approaches to learning. one is a zero-shot setting, where the goal is to generalize to prompts for which no training examples are given. prompts can also be used in a few-shot setting, in which a model is either (1) trained on prompted examples of the target task via gradient updates, or (2) priming (i.e. in-context learning), in which labeled examples are included in an input sequence in order to prime models to make predictions without gradient updates @xcite . promptsource was originally designed for zeroshot learning, so it emphasizes explicit task instructions and no priming examples. if needed, users can extend promptsource for few-shot learning (e.g., as done in @xcite . systems for annotating data most work on collecting annotations has focused on labels and other annotations at the level of individual examples @xcite . gate @xcite ) was an early system for annotating text, and includes support for many data types such as labels and entity tags. since then, many webbased systems for annotating text have been developed @xcite @xcite @xcite @xcite . other systems support collaboration among multiple annotators @xcite . more recently, many annotation systems have begun to incorporate learned models to improve workflow, using techniques such as active learning @xcite and example recommendation @xcite . these systems are possible because the annotations to be collected are labels, for which metrics like inter-annotator agreement and model confidence are available. there has also been some work on collecting annotations other than labels. alvisae @xcite and treeannotator @xcite support creating ontologies and other structured annotations. prompts differ from these annotations in that they are semi-structured functions, requiring new tools for developers. ## system design and workflow creating prompts differs from other types of data collection and annotation. we focus on three challenging aspects on which prompting differs from traditional nlp annotation: ## prompting language a key design decision is the format for prompts. previous works on prompting tended to use code for specifying each prompt. we experimented with this format and found a trade-off between expressivity and explicit structure. on one side, a maximally expressive format such as pure python code would let users write complex programs to manipulate the semi-structured examples into prompted examples. however, interpreting and analyzing these programs becomes difficult. this difficulty limits downstream manipulation and analysis of the prompts, for example for possible future work on automatic prompt augmentation. on the other side, a maximally structured format, such as rule-based generation, limits the kinds of prompts that users can create. we found it infeasible to enumerate types of rules sufficient for the wide range of tasks and data formats for which we wanted prompts. we therefore settled on a middle ground between the two: a templating language. specifically, we use the jinja2 templating engine, foot_0 originally designed for producing web markup. users write templates as prompts with placeholders, such as if {{premise}} is true, is it also true that {{hypothesis}}? ||| {{entailed}}. the separator ||| denotes the break between the conditioning text and the desired completion. placeholders refer to fields in the underlying example (represented as a python dict by datasets @xcite ). users also have access to jinja's built-in functions, such as manipulating strings and structured data. for each prompt, prompted examples are created by applying the prompt to all examples in the corresponding dataset. while jinja is a complete programming language, our review guidelines encourage simple functions with minimal additional logic (see figure 3 and 4 for example). during the development of promptsource, we found that a few idioms were particularly useful. first, not all templates are applicable to all examples in a dataset. users can wrap templates in jinja's built-in conditional statements, and any example that results in an empty prompted example is simply skipped. second, many examples can be used to make multiple training instances, such as a question that has multiple valid answers. we therefore added a choice function that selects an element from a list in a way that can be controlled during dataset generation, such as picking a random element using a seeded random number generator or generating different prompts for each combination of elements in the template. third, many tasks such as classification and binary question answering have a small set of possible valid completions, and it is common to make predictions for these tasks by scoring only the valid completions and returning the highest one @xcite @xcite . users therefore can list the valid completions in a separate field and access them as a list in their prompts (displayed as answer choices in figure 3 ). these completions are then explicitly available when evaluating predictions for these prompted examples. the promptsource system is designed to enable prompt creators to view data (s1), write prompts in a standard format (s2, s3, and s4), and verify that their templates work correctly (s5). we implemented a lightweight interface for the tool in streamlit 2 so that users could download, run locally in a web browser, and then upload their results to a central repository. testing iterations of the interface on pilot template-writing tasks, we converged on three views for the interface. v1: browse this view (figure 2 ) lets users inspect datasets before creating prompts (s1). once prompts are created, they can select prompts and browse the prompted examples generated by them (s5). the original example is viewed side-by-side with the resulting prompted example, with the substituted text highlighted to distinguish from text hard-coded in the template. users can quickly scroll through many examples, verify the behavior of their prompt, and return to the sourcing view if changes are needed. v2: sourcing this view (figures 3 and 4 ) allows users to select a dataset to prompt, browse examples from that dataset in the form of tables, and enter a prompt for that dataset. as the user writes their template (s2, s3, and s4), every time they save it, the output of the template applied to the current example is displayed next to the editor. we also collect metadata like a name for the template, and a reference for any bibliographic information or rationale for the template. v3: helicopter this view (figure 5 ) allows users to see what datasets are available for writing templates and how many are written for each, to prioritize user attention. this view is particularly useful for moving between datasets and for the prompt reviewers (s5). guidelines apply to the combination of a template (a function that maps an example into an input/target pair in natural language) and a set of metadata about the template. the most important constraint we imposed for a template to be valid is that it is formulated in natural language (both for the input and the target). we forbid the use of non-natural language prompts such as pure code. each prompt should clearly state what task should be solved, in a way a non-specialist adult can understand. we found this guideline strikes a good balance between freedom and expressivity in the wording of the prompts on one side and short generic prompts on the other side. in early experiments, we found that user-written prompts that did not explicitly state the possible valid completions tended to perform worse in experiments than their counterparts in which the possible valid completions were listed. we encouraged prompt authors to explicitly state the valid outputs in some of their prompts. in addition, when working with training prompts that include target text, we found it useful to remove variations on the target format that led to spurious ambiguity. for instance, the target template should only contain the answer to the task. it should not contain any extra text such as "the answer is ...", which can be equivalently moved to the input template. one of the research question we hope to enable with promptsource is whether the diversity of the prompt formulation during training leads to models that are more robust to the prompt formulation at test time. therefore, we encouraged prompt authors to create between 5 and 10 (or more) prompts per dataset while varying the prompt formulation. for a given dataset, authors produce multiple prompts per example, sometimes for task formulations that differed from the original dataset. for instance, for question answering dataset, one prompt can ask to extract the answer to a given question from a given passage, while a second prompt can ask to generate a potential question given an answer and a passage. as part of the community process and to facilitate future research, promptsource asks prompt authors to include additional metadata for each prompt. metadata fields include a name for the prompt, a reference to the paper it was extracted from (or any relevant explanation), whether the prompt expresses the task originally intended by the dataset, the valid outputs (if relevant), whether the input template states the valid outputs, and possible metrics to evaluate the prompted examples. these can be used in future systems to evaluate how the style and structure of prompts leads to different downstream results. ## case studies a system for creating, maintaining, and using prompts is a key tool for supporting the emerging research area of prompting in a standardized and reproducible manner. we highlight three recent research projects for which promptsource was a key resource. massively multitask prompted training @xcite study the question of zero-shot behaviors in large language models and ask whether zero-shot generalization can be induced by training a language model on a massively multitask mixture of prompts. to test this question, they use promptsource to create diverse prompts for a large collection of nlp datasets. their training and evaluation prompts are a subset of p3. this work demonstrates that promptsource allows training a language model on a massively multitask mixture of prompted datasets and evaluating the ability of models trained with such a procedure to perform unseen tasks. multilingual prompting lin et al. ( 2021 ) study the zero-and few-shot learning abilities of an multilingual autoregressive language model trained on 30 languages. in particular, they are interested in the cross-lingual generalization of such models and benchmark a variety of tasks in multiple languages. promptsource allows using a massive set of highquality english prompts. moreover, the english prompts serve as support to create prompts in other languages (through either machine or human translation). priming (in-context learning) @xcite study improving models' few-shot priming performance by first fully training a model (with gradient updates) on a multitask mixture formatted with priming examples. they find that incorporating templates from p3 significantly further improves performance compared to training on priming examples alone. although promptsource was not originally designed for this specific form of prompting, users were able to easily use p3's template collection and the templating language for their own priming methods. ## conclusion promptsource is an open-source system for creating, sharing, and using natural language prompts and addresses the need for new collaborative and centralized tools to support the emerging research around prompting. the tool is designed to answer three key needs: a flexible template language, a suite of tools for prompt management, and community-driven quality standards. as of january 2022, promptsource includes a growing collection of 2,000 public prompts for roughly 170 datasets, and has already been an instrumental resource for multiple recent research projects.