Prompterator: Iterate Efficiently towards More Effective Prompts (2023)

Abstract: With the advent of Large Language Models (LLMs) the process known as prompting, which entices the LLM to solve an arbitrary language processing task without the need for finetuning, has risen to prominence. Finding well-performing prompts, however, is a non-trivial task which requires experimentation in order to arrive at a prompt that solves a specific task. When a given task does not readily reduce to one that can be easily measured with well established metrics, human evaluation of the results obtained by prompting is often necessary. In this work we present prompterator, a tool that helps the user interactively iterate over various potential prompts and choose the best performing one based on human feedback. It is distributed as an open source package with out-of-the-box support for various LLM providers and was designed to be easily extensible.

## introduction the standard approach to solving language tasks using machine learning models has been to assume a train-test/metric setup, in which a task is well defined in advance, a dataset relevant for the task is gathered and split into a set of data that is used for training a specific model and optimizing its hyperparameters in a supervised way (train) and testing its performance (test), as well as one or multiple metrics that are used to provide a summary of the model's performance on this dataset. in practice, however, it is very difficult for many language tasks to be carefully designed in advance, in order for the dataset to be collected and an appropriate metric to be defined. to alleviate this issue, an alternative approach called prompting or prompt engineering rose to prominence recently. in it, the user provides a description of the task in along with instructions for the model to transform the provided input into a specific output. this is provided in a natural language (for instance english) and is also referred to as prompt or a prompt template, as part of the prompt is interpolated with each specific input. for instance, "provide a paraphrase of the following sentence: {text}", where {text} would be replaced with a particular input text sentence, is an example of a prompt that could be directly used for the sentence paraphrasing task, without the need to gather any train or test data. to aid the model in solving specific tasks, the prompt can also contain a list of training examples. this approach has risen to prominence with the introduction of large language models (llms), such as gpt-3 @xcite , gopher @xcite , opt @xcite and bloom @xcite . these models were trained on vast amounts of training data, their size often exceeds a hundred billion parameters and they demonstrate strong generalization capability in zero-shot or few-shot learning, in which they are able to learn to perform a new task based on the prompt alone, without requiring any parameter updates in the underlying model. it is hence the combination of a model and a prompt that yields the ability to solve a specific language task. finding an appropriate prompt for a specific model thus becomes of critical importance, as the choice of a specific prompt has an outsized impact on the final performance @xcite @xcite . in this work, we present prompterator, an integrated development environment for llm prompts. it provides a web-based interface that allows the user to interactively iterate over various prompt variants, evaluate their performance and save the evaluation data for future use. the application works locally and uses standardized json files that can be easily version controlled, allowing for decentralized team collaboration. it supports various llm providers such as openai, anthropic, google vertex ai, aws bedrock, cohere, as well as huggingface transformers @xcite out of the box. it provides an easily programmable interface for integrating other models in the future. unlike other related works in the area, it does not make any assumptions about the task itself, making it a suitable choice for any language processing task where llms can be reasonably used and human evaluation is appropriate. ## system description and key features in this section, we provide a walkthrough of prompterator's functionality from the point of view of a potential user whose aim is to arrive at a prompt that is capable of solving a specific task of their interest. to make the walkthrough a bit more concrete, let us consider the example of jane, a data scientist at a company that builds a q&a platform, who was tasked with finding out whether chatgpt (that is, gpt-3.5-turbo) would be capable of automatically shortening the questions coming to the q&a sessions after the interest in such a feature has been validated via user research calls. ## experiment to evaluate the effectiveness of prompterator we conduct a user study to investigate the time it takes to perform each part of the prompt engineering lifecycle (setup, predicting, evaluation, prompt updating) in prompterator as well as two other prompt-engineering setups. the baseline setup is a code-free pipeline consisting of microsoft excel used for data labeling and openai playground for generating model responses. spellbook is a saas prompt-engineering ide which, just like prompterator, supports multiple steps of the prompt-engineering pipeline such as uploading and processing of datasets, generating model responses, labeling, and evaluation. to the best of our knowledge, all other publicly available prompt-engineering ides focus on text classification, with limited to no support for text-to-text tasks. ## conclusion we present prompterator, a lightweight prompt engineering ide, which is capable of covering the whole lifecycle of finding an appropriate prompt for a given language task. prompterator has proven to be significantly faster and simpler to use than its alternatives across multiple sub-tasks of the prompting pipeline. by opensourcing it under the terms of the mit license we hope that it will make the process of prompt engineering singificantly more efficient in the future.