P rompt2 M odel: Generating Deployable Models from Natural Language Instructions (2023)

Abstract: Large language models (LLMs) enable system builders today to create competent NLP systems through prompting, where they only need to describe the task in natural language and provide a few examples. However, in other ways, LLMs are a step backward from traditional special-purpose NLP models; they require extensive computational resources for deployment and can be gated behind APIs. In this paper, we propose Prompt2Model, a general-purpose method that takes a natural language task description like the prompts provided to LLMs, and uses it to train a special-purpose model that is conducive to deployment. This is done through a multi-step process of retrieval of existing datasets and pretrained models, dataset generation using LLMs, and supervised fine-tuning on these retrieved and generated datasets. Over three tasks, we demonstrate that given the same few-shot prompt as input, Prompt2Model trains models that outperform the results of a strong LLM, gpt-3.5-turbo, by an average of 20% while being up to 700 times smaller. We also show that this data can be used to obtain reliable performance estimates of model performance, enabling model developers to assess model reliability before deployment. Prompt2Model is available open-source at https://github.com/neulab/prompt2model. Our demo video is posted at youtu.be/LYYQ_EhGd-Q .

## introduction traditionally, building an nlp model from scratch has been a substantial undertaking. an nlp practitioner seeking to solve a new problem would need to define their task scope, find or create data that specifies the intended system behavior, choose a suitable model architecture, train the model, assess its performance through evaluation, and then deploy it for real-world usage @xcite . llms like gpt-3 @xcite offer a lighter-weight paradigm for nlp system construction through "prompting" @xcite . practitioners can now write a prompt specifying the intended system behavior (optionally with a few demonstrations), and ask an llm to generate a desired output via text completion. this makes it possible to prototype nlp systems rapidly for a variety of applications without writing a single line of code @xcite . however, there is still a gap between proofof-concept prototyping -showing llms can be prompted for a particular task -and practical deployment. prompting llms can be expensive as they require either a significant amount of computing or access to commercial apis, and their reliance on the input prompt quality makes them unstable compared to trained models @xcite . because practitioners usually lack annotated data, it is also more challenging for them to evaluate or debug their systems before deployment @xcite . additionally, practitioners have expressed concerns about the high serving cost and slow prediction time associated with using llms @xcite , and those working in high-stakes domains cannot rely on commercial llm apis due to privacy concerns. in this work, we present prompt2model, a system that retains the ability to specify system behavior in a light-weight way through prompting, while still resulting in a deployable specialpurpose model, maintaining all the advantages thereof. prompt2model is designed as an automated pipeline that extracts essential task information from users' prompts and then automatically collects and synthesizes task-specific knowledge through three channels: prompt2model is designed to support different instantiations of each of these components. we provide a reference implementation using a gpt-3.5-turbo-based dataset generator, a dataset retriever based on datafinder @xcite , and a bm25-based model retriever. we evaluate three tasks covering both traditional nlp benchmarks and novel applications and find that, empirically, prompt2model sometimes produces small models that outperform gpt-3.5-turbo using the same prompt. on 2 of these 3 tasks, we @xmath0.5-turbo, despite our finetuned model being up to 700 times smaller. we also find that prompt2model can generate effective evaluation datasets. prompt2model can serve the following purposes for the community: ## prompt2model framework prompt2model provides a framework to automate machine learning pipelines: data collection, model training, evaluation, and deployment. we illustrate our automated pipeline in figure 2 . at the core is our automatic data collection system, leveraging dataset retrieval and llm-based dataset generation to obtain labeled data. we then retrieve pretrained models and finetune them on the training datasets. finally, we evaluate trained models on the test datasets and create a web ui for interaction. our general-purpose method is designed to be modular and extensible; each component can be implemented differently or disabled by a practitioner. we give an overview of our framework, then in section 3 we describe our reference implementation. prompt parser as the primary input to our system, users provide prompts similar to those used for llms. these prompts comprise an instruction and, optionally, a few demonstrations of the anticipated behavior. while this open-ended interface is convenient for users, end-to-end ml pipelines may benefit from a prompt parser that processes this input, such as segmenting the prompt into an instruction and individual demonstrations. dataset retriever given a prompt, we first try to discover existing manually annotated data that support users' task descriptions. there are several design decisions for the dataset retriever: 1. what datasets to search against? 2. how to index datasets for search? 3. which dataset columns are needed for the user's task, and which columns should be ignored? we use datafinder @xcite ) in our implementation, described in §3.2. dataset generator not all conceivable tasks have any existing relevant annotated data. to support a wider range of tasks, the dataset generator synthesizes training data from user-specific requirements parsed by the prompt parser. this presents challenges related to cost efficiency, speed, diver- the children's book test prompt + few examples dataset retriever dataset generator generated training set generated test set retrieved dataset model trainer evaluation performance estimate interactive demo input parser prompt spec model retriever retrieved model " answer questions given context from a relevant wikipedia article. examples: <qa @xmath1...] @xmath2: 58.9 em: 61.5 bertscore: 94.0 trained model sity, and quality control. our suggested solution to these challenges is described in §3.3. model retriever besides training data, we must identify an appropriate model to finetune. we cast this as a retrieval problem against model descriptions and metadata such as popularity or tasks supported. the reference implementation of our model retriever, described in §3.4, searches against models on hugging face @xcite . training given retrieved and generated datasets and a pretrained model, we use a model trainer to finetune the model on a subset of the data. evaluation after training models on a portion of the retrieved and generated datasets, we give the remaining data to an model evaluator module. selecting the correct metrics for an arbitrary task is a difficult problem. we describe our suggested strategies for task-agnostic evaluation in §3.6. ## reference implementation prompt2model is designed modularly to support customization of each component in our framework (described in §2), but we have provided a reference implementation to enable immediate adoption. ## experimental setup tasks as a proof-of-concept, we test our system's ability to learn a model for three tasks: • machine reading question answering: we use squad @xcite as ground truth to evaluate the setting where pretrained models and training datasets are plentiful. llm baseline a primary goal of our work is to train small models that can match or outperform llms. to measure success towards this goal, we report the performance of gpt-3.5-turbo on each benchmark. we provide gpt-3.5-turbo foot_1 the same instruction and demonstrations provided to prompt2model, for fair comparison. ## discussion and conclusion we propose prompt2model, a framework that automatically constructs task-specific models using only natural language prompts. our proof-ofconcept experiments show that, despite using a similar simple interface like llms, prompt2model delivers small yet accurate models and its generated datasets can be used to estimate real-world performance. prompt2model's extensible and modular design makes it a platform for advancing model distillation, dataset generation, synthetic evaluation, dataset retrieval, and model retrieval. we believe our prompt2model framework can inspire various novel research questions. these questions may include how much data should we generate for downstream model training and how diverse should it be? how do we effectively mix retrieved and generated data to achieve comple-mentary strengths (e.g. using dataset generation to focus on inputs that the retrieved dataset fails to cover)? since users may struggle to articulate their needs, future extensions should address humanin-the-loop correction by enabling either iterative editing of prompts or post-hoc fixing of data and models when the task metadata extraction and generated data do not align with user intentions. we invite the community to contribute novel implementations of various components in our framework.