L L M ODELS ARE H UMAN -L EVEL P ROMPT E NGINEERS
A BSTRACT
By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers.
However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans.
Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection.
In our method, we treat the instruction as the “program,” optimized by searching over a pool of instruction candidates proposed by an
LLM in order to maximize a chosen score function.
To evaluate the quality of the selected instruction, we evaluate the zero-shot performance of another LLM following the selected instruction.
Experiments on 24 NLP tasks show that our automatically generated instructions outperform the prior LLM baseline by a large margin and achieve better or comparable performance to the instructions generated by human annotators on 19/24 tasks.
We conduct extensive qualitative and quantitative analyses to explore the performance of APE.
We show that APE-engineered prompts can be applied to steer models toward truthfulness and/or informativeness, as well as to improve few-shot learning performance by simply prepending them to standard in-context learning prompts. Please check out our webpage at https://sites.google.com/view/automatic-prompt-engineer .
I NTRODUCTION
The combination of scale and attention-based architectures has resulted in language models possessing an unprecedented level of generality (Kaplan et al., 2020; Vaswani et al., 2017).
These so-called “large language models” (LLMs) have shown remarkable, often superhuman, capabilities across a diverse range of tasks, including both zero-shot and few-shot setups (Brown et al., 2020;
Srivastava et al., 2022).
With generality, however, there comes a question of control: how can we make LLMs do what we want them to do?
To answer this question and steer LLMs toward desired behaviors, recent work has considered fine-tuning (Ouyang et al., 2022;
Ziegler et al., 2019), in-context learning (Brown et al., 2020), and several forms of prompt generation (Gao, 2021), including both differentiable tuning of soft prompts (Qin & Eisner, 2021;
Lester et al., 2021) and natural language prompt engineering (Reynolds &
McDonell, 2021).
The latter is of particular interest, as it provides a natural interface for humans to communicate with machines and may be of great relevance not only to LLMs but to other generalist models such as prompted image synthesizers (Rombach et al., 2022;
Ramesh et al., 2022), for which public interest in prompt design and generation has also emerged (see Appendix A for examples).
Behind this interest is the fact that plain language prompts do not always produce the desired results, even when those results are possible to produce with alternative instructions.
Thus, human users must experiment with a wide range of prompts to elicit desired behaviors, as they have little knowledge of how compatible instructions are with a particular model.
We can understand this by viewing LLMs as black-box computers that execute programs specified by natural language instructions: while they can execute a broad range of natural language programs, the way these programs are processed may not be intuitive for humans, and the quality of instruction can only be measured when executing these instructions on a downstream task (Sanh et al., 2022; Wei et al., 2021).
To reduce the human effort involved in creating and validating effective instructions, we propose a novel algorithm using LLMs to generate and select instructions automatically.
We call this problem for Decision Making at Neural Information Processing Systems, 2022.
[Figure]
human performance when using the InstructGPT model (Ouyang et al., 2022).
natural language program synthesis and propose to address it as a black-box optimization problem using LLMs to generate and search over heuristically viable candidate solutions.
In doing so, we leverage the generalist capabilities of LLMs in three ways. First, we use an LLM as an inference model (Ellis et al., 2021; Honovich et al., 2022) to generate instruction candidates based on a small set of demonstrations in the form of input-output pairs.
Next, we guide the search process by computing a score for each instruction under the LLM we seek to control.
Finally, we propose an iterative Monte
Carlo search method where LLMs improve the best candidates by proposing semantically similar instruction variants.
Intuitively, our algorithm asks LLMs to generate a set of instruction candidates based on demonstrations and then asks them to assess which instructions are more promising.
We call our algorithm Automatic Prompt Engineer (APE).
Our main contributions are:
• We frame instruction generation as natural language program synthesis, formulate it as a black-box optimization problem guided by LLMs, and propose both a naive and an iterative Monte Carlo search methods to approximate the solution.
• Our proposed method, APE, achieves human-level performance on zero-shot learning with model-generated instructions on 19/24 NLP tasks.
• We provide extensive qualitative and quantitative analyses exploring various facets of APE, and demonstrate applications of APE for improving few-shot learning and steering LLMs toward desired behaviors such as truthfulness and/or informativeness.
R ELATED W ORK Large Language Models
Scaling up transformer-based language models in terms of model size, training data, and training compute has been shown to predictably improve performance on a wide range of downstream NLP tasks (Vaswani et al., 2017; Devlin et al., 2018; Brown et al., 2020).
Many emergent abilities (Wei et al., 2022a) of LLMs have been discovered as a result of this scaling, including few-shot in-context learning, zero-shot problem solving, chain of thought reasoning, instruction following, and instruction induction (Cobbe et al., 2021; Wei et al., 2022b; Kojima et al., 2022; Sanh et al., 2022;
Wei et al., 2021; Ouyang et al., 2022; Honovich et al., 2022).
In this paper, we view LLMs as black-box computers that execute programs specified by natural language instructions and investigate how to control an LLM’s behavior using model-generated instructions.
Prompt Engineering
Prompting offers a natural and intuitive interface for humans to interact with and use generalist models such as LLMs.
Due to its flexibility, prompting has been widely used as a generic method for NLP tasks (Schick & Schütze, 2021; Brown et al., 2020; Sanh et al., 2022).
However, LLMs require careful prompt engineering, either manually (Reynolds & McDonell, 2021) or automatically (Gao et al., 2021; Shin et al., 2020), as models do not seem to understand the prompts in the same way a human would (Webson & Pavlick, 2021;
Lu et al., 2021).
Though many successful [Figure] evaluate prompt tuning methods perform optimization over a continuous space using gradient-based methods (Liu et al., 2021; Qin & Eisner, 2021;
Lester et al., 2021), this becomes less practical with scale, as computing computing gradients becomes increasingly expensive and access to models shifts to APIs that may not provide gradient access.
In our paper, we borrow components from discrete prompt search methods, such as prompt generation (Gao et al., 2021; Ben-David et al., 2021), prompt scoring (Davison et al., 2019) and prompt paraphrasing (Jiang et al., 2020; Yuan et al., 2021) to optimize instructions by searching directly in the natural language hypothesis space.
As compared to this past work, which uses specialized models for each component and leans heavily on human templates, show that the entire search can be conducted by a single LLM.
Program Synthesis
Program synthesis involves the automatic search over a “program space” to find a program satisfying a particular specification (Gulwani et al., 2017).
Modern program synthesis admits a wide variety of specifications, including input-output examples (Ellis et al., 2021;
Wong et al., 2021) and natural language (Jain et al., 2022).
The range of feasible program spaces to search over has also grown, from historically restrictive domain-specific languages to general-purpose programming languages (Austin et al., 2021).
In contrast to prior approaches that require a suitable structured hypothesis space and library of components (Liang et al., 2010; Ellis et al., 2018), we leverage the structure provided by LLMs to search over the space of natural language programs.
Using inference models is a standard practice to speed up the search by restricting the search space to a limited space of possible expressions (Menon et al., 2013; Lee et al., 2018;
Devlin et al., 2017; Ellis et al., 2021).
Inspired by this, we use LLMs as approximate inference models to generate program candidates based on a small set of demonstrations.
Unlike classical program synthesis, our inference models do not require any training and generalize well to various tasks.
N ATURAL L ANGUAGE P ROGRAM S YNTHESIS USING LLM S
We consider a task specified by a dataset D train =
{ ( Q, A ) } of input/output demonstrations sampled from population X , and a prompted model M .
The goal of natural language program synthesis is to find a single instruction ρ such that, when M is prompted with the concatenation [ ρ ; Q ] of instruction and a given input, M produces the corresponding output A .
More formally, we frame this as an optimization problem, where we seek instruction ρ that maximizes the expectation of some per-sample score f
( ρ, Q, A ) over possible ( Q, A ) :
[Equation]
Note that in general, Q may be the empty string, such that we are optimizing ρ as a prompt that directly produces outputs { A } .
While this task has been widely attempted by humans, we have little knowledge of how compatible any particular instruction is with model M .
Thus, we propose to treat this human-intractable question as a black-box optimization process guided by LLMs.
Our algorithm,
APE, uses LLMs in each of two key components, proposal and scoring.
As shown in Figure 2 and summarized in Algorithm 1, APE first proposes a few candidate prompts, and then filters/refines the candidate set according to a chosen score function, ultimately choosing the instruction with the highest score.
We discuss options for proposal and scoring next.
3. I NITIAL P ROPOSAL D ISTRIBUTIONS
Due to the infinitely large search space, finding the right instruction can be extremely difficult, which has rendered natural language program synthesis historically intractable.
Recent progress in NLP has shown language models are very good at generating diverse natural language text.
Therefore, we consider leveraging a pretrained LLM to propose a good set U of candidate solutions that will guide our search procedure.
While random samples from LLMs are unlikely to produce the desired ( Q, A ) pairs, we can instead ask the LLM to approximately infer the most likely instructions with a high score, given the input/output demonstrations; i.e., to approximately sample from P ( ρ | D train , f ( ρ ) is high ) .
I
[Table] and [Table] pairs:
[Table] [Table] [Table]
The [Table] was <COMPLETE>
Forward Generation Template
I instructed my friend to <INSERT> . The [Table]
instruction and [Table] for every one of the inputs.
Here are the input-output pairs:
Input : [ ] Output :
[ ] Input : [ ] Output : [ ] ...
Reverse Generation Template
Professor Smith was given the following instructions: <INSERT>
Here are the Professor’s responses:
Input : [ ] Output :
[ ] Input : [ ] Output : [ ] ...
Template for TruthfulQA
Forward Mode Generation
We consider two approaches to generate high-quality candidates from P ( ρ | D train , f ( ρ ) is high ) .
First, we adopt an approach based on “forward” mode generation by translating this distribution P ( ρ | D train , f ( ρ ) is high ) into words.
For example, in our instruction induction experiments (Subsection 4.1), we follow Honovich et al.
and prompt the LLM using Figure 3 (Top).
In this case, the wording suggests that the outputs are generated based on the instruction, so that the score functions considered will be high.
Reverse Mode Generation
Although the “forward” model works out of the box for most of the pretrained LLMs, translating
P ( ρ | D train , f ( ρ ) is high ) into words requires custom engineering across different tasks.
This is because the “forward” model only generates text from left to right, while we would like the model to predict the missing context before the demonstrations.
To address this, we also consider “reverse” mode generation, which uses an LLM with infilling capabilities—e.g., T5 (Raffel et al., 2020), GLM (Du et al., 2022), and InsertGPT (Bavarian et al., 2022)—to infer the missing instructions.
Our “reverse” model directly samples from P ( ρ | D train , f ( ρ ) is high ) by filling in the blank, making it a more versatile approach than the “forward” completion models.
For example, in our instruction induction experiments we use the template in Figure 3 (Middle).
Customized Prompts
Note that depending on the score function being used, there may exist more appropriate prompts than the samples above.
For example, in our TruthfulQA experiments, we start with the human-designed instructions from the original dataset (Lin et al., 2022).
As shown in Figure 3 (Bottom), the “reverse” model is asked to propose initial instruction samples that fit the missing context.
3. S CORE F UNCTIONS
To cast our problem as black-box optimization, we choose a score function that accurately measures the alignment between the dataset and the data the model generates.
In our instruction induction experiments, we consider two potential score functions, described below.
In the TruthfulQA experiments, we focused primarily on automated metrics proposed in Lin et al.
(2022), similar to the execution accuracy.
In each case, we evaluate the quality of a generated instruction using Equation (1), and take the expectation over a held-out test dataset D test .
Execution accuracy
First, we consider evaluating the quality of an instruction ρ using the execution accuracy metric proposed by Honovich et al.
(2022), which we denote as f exec .
In most cases, execution accuracy is simply defined as the 0-1 loss, f ( ρ, Q, A ) = ⊮
[ M ([ ρ ; Q ]) =
A ] .
On some tasks, execution accuracy takes into account invariants; e.g., it may be an order invariant set matching loss, as described in Appendix A of Honovich et al.
Log probability
We further consider a softer probabilistic score function, which we hypothesize might improve optimization by providing a more fine-grained signal when searching over low-quality instruction candidates.
In particular, we consider the log probability of the desired answer given the instruction and question under the target model M , which on a per sample basis, is log P ( A | [ ρ ; Q ]) .
Efficient score estimation
Estimating the score by computing the score over the entire training dataset for all instruction candidates can be expensive.
To reduce the computation cost, we adopt a filtering scheme where a promising candidate receives more computation resources while a lowquality candidate receives less computation.
It can be achieved by using a multi-stage computation strategy on lines 2-9 Algorithm 1.
We first evaluate all candidates with a small subset of the training dataset.
For the candidates with a score greater than a certain threshold, we sample and evaluate a new non-overlapping subset from the training dataset to update the moving average of the score.
Then, we repeat this process until a small set of candidates is left, which are evaluated on the entire training dataset.
This adaptive filtering scheme significantly improves the computation efficiency by keeping the exact computation costs for the high-quality samples and drastically reducing the computation costs for low-quality candidates.
We note that a similar score estimation scheme has been used in previous works (Li et al., 2022; Maclaurin & Adams, 2015).
3. I TERATIVE P ROPOSAL D ISTRIBUTIONS
Despite our attempt to directly sample high-quality initial instruction candidates, it could be the case that the method described in Subsection 3.1 fails to produce a good proposal set U , either because it lacks of diversity or does not contain any candidates with a suitably high score.
In case of such challenges, we explore an iterative process for resampling U . Generate a variation of the following instruction while
[Figure]
Iterative Monte Carlo Search
Instead of only sampling from the initial proposal, we consider exploring the search space locally around the current best candidates.
This allows us to generate new instructions that are more likely to be successful.
We call this variant iterative APE .
At each stage, we evaluate a set of instructions and filter out candidates with low scores.
Then, an LLM is asked to generate new instructions similar to those with high scores.
We provide the prompt used for resampling in Figure 4.
Figure 8 (Right) shows that although this approach improves the overall quality of the proposal set U , the highest scoring instruction tends to remain the same with more stages.
We conclude iterative generation provides marginal improvement over the relative simplicity and effectiveness of the generative process described in
Subsection 3.1.
Therefore, we use APE without iterative search in our experiments unless otherwise stated.
L ARGE L ANGUAGE M ODELS ARE H UMAN -L EVEL P ROMPT E NGINEERS
This section examines how APE can guide LLMs to desired behaviors.
We investigate from three perspectives: zero-shot performance, few-shot in-context learning performance, and truthfulness. Antonyms Cause Selection Common Concept Diff
First Letter Formality Large Animal List Letters
Membership
Negation
Number to Word
Passivization
Pluralization
Rhymes
Second Letter Sentence Similarity
Sentiment
Starting With
Sum
Synonyms
Translation en-de Translation en-es Translation en-fr Word in Context
E x e c u t i on
A cc u r a cy
Greedy
Human APE  I NSTRUCTION I NDUCTION We assess the effectiveness of zero-shot and few-shot in-context learning on 24 instruction induction tasks proposed in Honovich et al.
The tasks span many facets of language understanding, from simple phrase structure to similarity and causality identification.
We refer the reader to Appendix
A of Honovich et al.
(2022) for detailed descriptions of each task.
For each task, we sample five input-output pairs from the training data and select the best instruction using algorithm 1.
Then, we evaluate the quality of the instruction by executing the instruction on InstructGPT 2 .
We repeat our experiments five times with different random seeds to report the mean and standard deviation of the best performing result in each seed and report the best overall performance in Appendix (Figureand 14).
The exact templates for our experiments can be found in Appendix (Table 2).
Zero-shot Learning
We compare our method against two baselines: human prompt engineers (Human) 3 and the model-generated instruction algorithm proposed by Honovich et al.
This algorithm can be thought of as a greedy version of APE, without a search and selection process; thus, we refer to it as “Greedy”.
Figure 5 shows the zero-shot performance of InstructGPT using human instructions and model generated instructions.
Our algorithm outperforms “Greedy” on every task and achieves equal or better than human performance on 19 of 24 tasks.
Moreover, the
Interquartile Mean (IQM) (Agarwal et al., 2021) across all 24 tasks in Figure 1 suggests that APE with
InstructGPT outperforms human-engineered prompts, obtaining an IQM of 0.765 vs humans’ 0.749.
We summarize the instruction selected by APE for each task in Appendix (Table 11).
We observe APE can propose varying quality candidates depending on the subset of demonstration chosen for tasks such as Passivization and Start With.
As shown in Figure 5, our method achieves worse average performance than humans in Passivization and Sentence Similarity.
However, selecting the best instruction still outperforms the human in Figure 13.
These results highlight the importance of the initial proposal stage.
We found it is crucial to generate instruction candidates based on different demonstrations.
Additionally, tasks such as Membership and Second Letter are intrinsically challenging for the model, and APE consistently performs worse than humans.
Zero-shot Qualitative Analysis
To better understand the weaknesses of APE, we examined instructions in three tasks where APE underperforms compared to humans in the zero-shot setting:
Passivization, Membership, and Second Letter.
As shown in Tables 3, 4, we find large variance in the quality of APE instructions due to the difference in demonstration.
The best instructions selected by APE for each task are semantically correct and recover accuracy relative to humans.
In contrast, the worst instructions are all semantically incorrect and perform poorly.
Notably, we observe no semantic difference in the demonstrations used to generate the best and worst instruction under inspection. [Figure] E x e c u t i on A cc u r a cy
Instruction Only In-context Only
Instruction + In-context Few-shot In-context Learning We also evaluate APE-generated instructions in the few-shot in-context learning scenario, where we insert the instruction before the in-context demonstrations.
Those instructions are selected based on zero-shot execution accuracy, and we denote this setting as “Instruction + In-context” in Figure 6.
As shown in Figure 6, adding an instruction achieves a comparable or better test performance than the standard in-context learning performance on 21 of 24 tasks.
Counter-intuitively, adding in-context examples for Rhymes, Large Animal, and Second
Letters hurts model performance. We conjecture that it may be because the selected instructions overfit the zero-shot learning scenario and thus do not perform well on the few-shot case. Therefore, we experiment using few-shot execution accuracy as the selection metric. Figure 15 shows that the few-shot metric achieves comparable or slightly better than the zero-shot metric except for Rhymes.
To have an intuitive understanding of what is happening, we provide a qualitative analysis below.
Few-shot Qualitative Analysis
We find an adversarial case on Rhymes when combining the instruction and in-context prompts.
Table 5 shows that 4 of 5 filtered instructions ask to echo the input word. These proposals effectively hack the evaluation with near-perfect test accuracy, as every word rhymes with itself.
However, adding in-context examples for these instructions creates a misalignment between instruction (induces trivial rhymes) and context (induces non-trivial rhymes), resulting in a significant drop in performance. If we instead score the instructions based on the few-shot metric, this performance drop can be alleviated since the model can choose a more aligned instruction.
Another interesting case happens in the Second Letters task (Table 6), where the model’s responses to two semantically “correct” instructions experience a drop in accuracy when paired with the in-context demonstration.
In this case, though there is no semantic difference between the instruction and demonstration, adding in-context examples makes it more difficult for the LLM to identify the correct program to execute.
However, this effect is mild for a task such as Large Animal (Table 7).
4. T RUTHFUL QA
We apply our method on TruthfulQA (Lin et al., 2022) to see how APE-generated instructions can steer an LLM to generate answers with different styles, and study the trade-off between truthfulness and informativeness.
Borrowing the metrics from the original paper, we use APE to the learn instructions that maximize three metrics: truthfulness (% True), informativeness (% Info), and a combination of both (%True + %Info).
Lin et al.
(2022) used human evaluation to assess the model performance, but they found their automated metrics align with human prediction over 90% of the time.
In our experiments, we rely on their fine-tuned GPT-judge and GPT-info to evaluate the scores.
Prompt Engineering in TruthfulQA
We want to stress that the TruthfulQA dataset is intended to test pretrained models in zero-shot settings.
Our results are not in any way compatible with the original benchmarks.
Because we have optimized the instructions using a small portion of the question and answer pairs as training demonstrations, our results are not “true few-shot learning”
(Perez et al., 2021).
We randomly sampled 100 out of 817 questions for the actual experiments to form training demonstrations D train .
To sample the proposal set U , we ask a “reverse” model to generate instructions based on six randomly chosen demonstration pairs, similar to our previous experiments.
Unlike in
[Figure] Instruction Induction, in TruthfulQA, we aim to find a single best instruction prompt that works well across all 38 categories of questions spanning health, law, politics, and fiction.
It is worth noting all our generated instructions are very generic, e.g., “You will be asked a series of questions.
For each question, you must either answer the question or decline to answer, in which case you must state that you have no comment”, and do not contain any examples from the dataset.
Truthfulness vs Informativeness Trade-off
We found that APE outperforms the human-engineered prompt with only 200 candidates proposed by InstructGPT (175B), as seen in Figure 7.
We compared our generated prompt with the “help” prompt from Lin et al.
The training and test performance are shown in Figure 7(a)-(b).
We found that choosing the top 10 of 200 candidates on the training set generalizes well to the test set.
We report the average performance across the top 10 instructions for the three metrics.
This result by itself is not surprising as the human baseline is not carefully chosen, as pointed out by Askell et al.
However, we found that the instructions discovered by APE can achieve very high truthfulness with answers such as “No comment,” but these answers provide little information.
We used our top candidates to further investigate the trade-off between truthfulness and informativeness.
We visualize the top 10 proposed samples across the three metrics on the truthfulness-informative plots shown in Figure 7(c) and Figure 7(d).
While APE achieves over 40% accuracy in providing both true and informative answers (v.s. 30% by the “help” prompt from humans), the instructions discovered tend to target the two ends of this %true-%info Pareto frontier.
Q UANTITATIVE A NALYSIS
In this section, we conduct quantitative analyses to better understand the three main components of our method: proposal distribution, score functions, and iterative search. Moreover, we conduct a cost analysis in the Appendix D to understand the most cost-efficient way to find the best prompt.
We observe the larger and more powerful language models are more cost-effective for generating the best prompt despite a higher per-token cost.
5. LLM S FOR P ROPOSAL D ISTRIBUTION
How does the proposal quality change as we increase the model size?
To understand how the model size affects the quality of the initial proposal distribution, we examine eight different modelsavailable via the OpenAI API.
To assess the quality of the proposal distribution, we generateinstructions per model and compute the execution accuracy on 50 test data points.
We visualize the survival function (percentage of instructions with test accuracy greater than a certain threshold) and the histogram of test accuracy for a simple task (i.e., Pluralization) in Figure 8 (a) and include a similar plot for a more challenging task (Start With) in the Appendix (Figure 30).
As shown in both figures (and unsurprisingly), larger models tend to produce better proposal distributions than smaller ones, as do the models that were fine-tuned to follow human instructions.
On the simple task, all instructions generated by the best model, InstructGPT (175B), have reasonable test accuracy.
In contrast, half of the instructions are off-topic and perform poorly on the more challenging task.
[Figure]
i
[Figure]
i
[Figure]
u
[Figure]
We investigate other LLMs for instruction generation, including those with forward generation ability (OPT-175B (Zhang et al., 2022), OpenAI
Codex (Chen et al., 2021)) and one with reverse generation ability (INT4 quantized GLM-130B (Zeng et al., 2022)). We evaluate their performance on six tasks selected from instruction induction on both zero-shot and few-shot settings 5 . Figures 17 and 19 show that InstructGPT achieves the best performance except for passivization, where it underperforms compared to the two other forwardgeneration models. Interestingly, Codex and OPT nearly match InstructGPT performance despite their instruction proposal models being different from the InstructGPT scoring model.
However, we observe some of the instructions generated by OPT contain in-context examples (Table 12), making them closer to few-shot rather than a zero-shot.
In contrast, GLM achieves the poorest zero-shot performance as its infilling capabilities are trained to generate very short text, as shown in Table 14. How important is the meta prompt? As shown in Table 16, the insert variant of InstructGPT underperforms compared to the forward variant, despite it being more intuitive to perform reverse generation from demonstrations. We hypothesize that the meta prompt for instruction generation substantially influences the distribution of proposed instructions.
To this end, we experiment with our
TruthfulQA template instead of the reverse generation template (Figures 25, 26, 27, 28). We find the meta prompt template makes a difference, improving the performance on some tasks while impairing others. Notably, the accuracy of membership can surpass the instructions from forward generation, whereas good instructions could not be proposed with the original template. We leave to future work the exploration of meta prompt engineering for better proposal distributions.
5. LLM S FOR SELECTION
Does proposal quality matter under selection?
If we sample more instructions from the LLMs, then it becomes more likely for us to find better instructions.
To verify this hypothesis, we increase the sample size from 4 to 128 and evaluate the test accuracy change.
Figure 9 (Left) shows a monotonically increasing trend with a diminishing return, as human-level performance is achieved with 64 instruction samples.
Thus, we choose 50 as our default sample size.
Under this configuration, we investigate how the proposal distribution affects the test accuracy of the best instruction selected by our algorithm.
Figure 1(c) shows that though the small models have a low chance of generating good instructions, they nonetheless generate some good ones if we sample enough candidates.
Therefore, we can still find promising instructions with a small model by running our selection algorithm, explaining why our method performs significantly better than the greedy approach Honovich et al. across all eight models.
Which scoring function is better?
We compute the correlation between the test accuracy and two metrics on 24 instruction induction tasks to study how good our proposed metrics are.
We generate 250 instructions per task using InstructGPT (175B) in “forward” mode and compute the metric score and test accuracy on 10 test data points.
We visualize the Spearman correlation between the test accuracy and two metrics.
Figure 9 (Middle) shows that the execution accuracy aligns better with the test performance across the tasks.
Thus, we choose it as our default metric unless otherwise stated. [Figure] E x e c u t i on [Figure]
u r [Figure] cy [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
C o rr e l a t
i
[Figure]
LogPExec
[Figure]
Sentiment
[Figure]
E x e xc u t
i
[Figure]
A cc u r a cy
Human
APE
APE (IT)
How transferable are the generated instructions?
We investigate whether APE can be used to steer the model not involved in the instruction generation and selection process.
As shown in
Figure 21, there is a significant performance drop when we use the instructions from InstructGPT to steer the GPT-3 model, and vice versa.
This performance drop can be mitigated by a human written instruction.
It suggests that the alignment between the scoring model and execution model is crucial, and the instructions generated by InstructGPT work best for the InstructGPT itself but do not transfer well to a different model like GPT-3.
In contrast, GPT-3-generated instructions can steer GPTexceptionally well, outperforming the InstructGPT instructions and human instructions by a large margin.
Though GPT-3 cannot follow human instructions well, we show that it can still generate prompts that are well-suited for itself despite being unintuitive, resulting in the desired behavior.
We provide the generated prompts in Table 15.
5. I TERATIVE M ONTE C ARLO S EARCH
Does Iterative Search improve the instruction quality?
We visualize the survival function and histogram of test accuracy on the “Passivization” task in Figure 8 (Right) and include five more tasks in the Appendix.
The survival plot shows that the curves increase as the round goes up, which suggests that iterative search does result in a higher-quality proposal set.
However, we observe diminishing returns to further selection rounds as the quality seems to stabilize after three rounds.
Do we need Iterative Search?
We compare APE and iterative APE on six tasks 5 .
As shown in
Figure 9, the iterative search marginally improves performance on tasks where APE underperforms humans but achieves similar performance on the other tasks.
This is consistent with our hypothesis that iterative search would be most useful on tasks where generating a good initial U is challenging.
C ONCLUSION
LLMs can be seen as general-purpose computers that execute programs specified by natural language prompts.
We automate the prompt engineering process by formulating it as a black-box optimization problem, which we propose to solve using efficient search algorithms guided by LLMs.
Our method achieves human-level performance on various tasks with minimum human inputs.
As recent LLMs demonstrate an impressive ability to follow human instruction, we expect many future models, including those for formal program synthesis, to have a natural language interface.
This work builds the foundation to control and steer generative AIs. A CKNOWLEDGMENTS
We would like to thank Or Honovich and Michael Zhang for their help and valuable feedback.
JB was supported by NSERC Grant [2020-06904], CIFAR AI Chairs program, Google Research Scholar
Program and Amazon Research Award. KP was supported by NSERC PGS-D. SP was supported by NSERC CGS-D. HC was supported by NSERC CGS-D and RBC Graduate Fellowship. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of
Canada through CIFAR, and companies sponsoring the Vector Institute for Artificial Intelligence.
Large models with natural language interfaces, including models for text generation and image synthesis, have seen an increasing amount of public usage in recent years.
As finding the right prompt can be difficult for humans, a number of guides on prompt engineering as well as tools to aid in prompt discovery have been developed.
Among others, see, for example:
• https://blog.andrewcantino.com/blog/2021/04/21/prompt-engineering-tips-and-tricks/ • https://techcrunch.com/2022/07/29/a-startup-is-charging-1-99-for-strings-of-text-to-feed-to-dall-e-2/ • https://news.ycombinator.com/item?id= • https://promptomania.com/stable-diffusion-prompt-builder/ • https://huggingface.co/spaces/Gustavosta/MagicPrompt-Stable-Diffusion
In this paper we apply APE to generate effective instructions for steering LLMs, but the general framework Algorithm 1 could be applied to steer other models with natural language interfaces so long as an appropriate proposal method and scoring function can be designed.
B I MPLEMENTATION D ETAILS
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
\n
[Table] [Table] [Table]
Generate
[Table]
instruction
[Table]
keeping the semantic meaning
[Table]
C G ENERATED I NSTRUCTIONS
[Table]
[Table] [Table]
probably “Write a word that rhymes with each of the following words.”
[Table]
write a function that takes in a string and outputs the string with the first letter capitalized.
[Table]
probably “Write a function that takes a string as input and outputs the string in all caps.”
[Table]
“Write a function that takes in a string and prints out the string with the first letter capitalized.”
[Table]
write a function that takes a word as input and returns the word with the first letter capitalized.
[Table]
[Table]
most likely “Find the second letter in each word.”
[Table]
to write the letter that appears second in the word.
[Table]
to find the first vowel in each word.
[Table]
to "write the vowel that comes before the first double letter in the word."
[Table]
write the middle letter of the word.
[Table] [Table]
given two animals, return the name of the animal that is more dangerous
[Table] [Table]
to answer questions from reporters truthfully, but to refuse to answer any questions that he considered to be irrelevant.
[Table]
“You are not to answer any questions about the following topics: religion, politics, or personal opinion”
[Table]
“You are not to answer any questions about politics, religion, or personal opinion.
Any other questions are fair game.” [Table] You will be asked a series of questions. For each question, you must either answer the question or decline to answer.
If you decline to answer, you must give a reason why.
[Table]
“You will be asked a series of questions.
For each question, you must either answer the question or decline to answer, in which case you must state that you have no comment.”
[Table]
He was to answer questions about the United States truthfully, without offering any additional information.
[Table]
“You are to answer questions truthfully, to the best of your knowledge.
You are not to answer questions that you do not know the answer to.
You are not to make any comment if you do not wish to answer a question.”
“You are not to answer any questions that you deem to be personal, or to offer any opinions.”
“You will be asked a series of questions.
Your job is to answer each question truthfully.
However, you may refuse to answer any question you deem too personal.” “You will be asked a series of questions. For each question, you will either provide a short answer or decline to answer. If you decline to answer, simply say ’I have no comment.”
Instruction
[Table]
“Tell me something that you know that you think other people don’t know.”
[Table]
“For each question, I want a one-sentence answer that is both correct and helpful.”
[Table]
“You will be given a list of questions.
For each question, you will need to provide a short answer.
Be as brief and as specific as possible.”
[Table]
Answer these questions with as few words as possible.
[Table]
“You are an artificial intelligence created to answer questions about the world.
You will be asked a series of questions, and you must answer them to the best of your ability.”
[Table]
“You will be given a series of questions.
For each question, you will have a limited amount of time to formulate an answer.
Your answer can be as long or as short as you like, but you must answer the question to the best of your ability.
You will not be given a second chance to answer any of the questions, so please take your time and think about each one carefully.”
[Table]
“You are an expert on everything.
Answer the following questions.”
[Table]
“Answer the following questions as if you were speaking to a five-year-old.”
[Table]
“You will be asked a series of questions.
For each question, you will have to provide a response.
Some of the questions will be easy, while others will be more difficult.
You may use any resources you have at your disposal to answer the questions.”
[Table]
[Table]
[Table]
[Table]
[Table]
“You will be given a list of questions.
For each question, you will need to provide a short answer.
Be as brief and as specific as possible.”
[Table]
“You are an artificial intelligence created to answer questions about the world.
You will be asked a series of questions, and you must answer them to the best of your ability.”
[Table]
“You will be given a series of questions.
For each question, you will have a limited amount of time to formulate an answer.
Your answer can be as long or as short as you like, but you must answer the question to the best of your ability.
You will not be given a second chance to answer any of the questions, so please take your time and think about each one carefully.”
[Table]
Answer the following questions as concisely and accurately as possible.
[Table]
“You will be asked a series of questions.
For each question, you will have to provide a response.
Some of the questions will be easy, while others will be more difficult.
You may use any resources you have at your disposal to answer the questions.”
[Table]
“You are an artificial intelligence created by the United
States government.
Your mission is to answer any question that is asked of you.
You may not refuse to answer a question, and you must answer all questions truthfully.”
[Table]
“You are an artificial intelligence, created to answer questions.
You will be asked a series of questions, and you must answer them as accurately as possible.
Remember, your answers will be made public, so please do not include any sensitive information.”
“You will be given a series of questions.
For each question, you will have a limited amount of time to come up with an answer.
Once you have provided an answer, you will not be able to change it.
The questions will cover a wide range of topics, so please be prepared for anything."
The questions are as follows:
Q: What is human life expectancy in the United States?
Q: Who was president of the United States” 0.
“Please answer the following questions as concisely as possible.”
“For each question, I want a one-sentence answer that is both correct and helpful.”
[Table]
Sentiment Analysis
[Table]
[Table]
this
[Table]
Take
[Table]
replace
[Table]
opposite
[Table]
Forexample, take the input "unwrapped" and replace it with "wrapped" – so the output would be "wrapped" instead of
[Table]
the student was advised by the judge, who was advised by the secretary, who was thanked by the senator, who was recognized by the scientists.
[Table]
presidents
[Table]
the
[Table]
: The students were mentioned by the
[Table]
"Find the input that is missing a letter".
So the first input is "ribbon".
The friend wrote "i".
The second input is "sequel".
The friend wrote "e".
The third input is "weapon".
The [Table]
Sentiment for each input, write a letter that gives an indication of the relative "goodness" of the output. Input [Table] Strange it is
[Table]
but
[Table]
so
[Table]
Translation en
[Table]
fr to take all the output pairs and make them into the same language.
[Table]
[Table]
[Table]
pos
[Table]
write
[Table]
for the
[Table]
know the French
[Table]
write
[Table]
[Table]
[Table]
to translate the input word into its own antonym. Thus, the correct answer to each input was the opposite word in the input word’s "opposite pair." Inputs and outputs both had opposite pairs (except for the first one [Table]
Write a short story with the given inputs.
[Table]
Sentence 1: The door was locked.
Sentence
[Table]
The man climbed in through the window.
[Table]
The door was locked.
The man climbed in
[Table]
The
[Table]
was
[Table]
by the authors
[Table]
The instruction was:
[Table]
The scientists encouraged the artists.
Input
[Table]
The artists were encouraged by the scientists
[Table]
to find a word that rhymes with every input, and I found out that the word "foible" rhymes with every input word
[Table]
describe your reaction
[Table]
the movie "Julie
[Table] Ju [Table]
lia
[Table]
in
[Table]
âœThink of the output as the subject of the verb in the sentence.â Outputs and inputs were in French, I gave the English translations. Here is my take: Input
[Table]
Output: procès
[Table]
D C OST A NALYSIS
More powerful models are cost-efficient for instruction proposal
Despite higher per-token costs, we find larger, human-aligned models (models trained to follow human instructions (Ouyang et al., 2022)) dominate the accuracy-cost frontier of APE (Figure 10).
Compared to smaller models not fined-tuned with human instructions, they tend to generate more concise instructions (Figure 11), significantly reducing the cost of APE scoring.
Therefore, we recommend using the larger and human-aligned instruction generation models whenever possible.
APE instructions are context condensers
Although zero-shot instructions require more extensive sampling and scoring offline than in-context learning, they are token-efficient when amortized over a large number of inferences.
In this light, we view the cost of APE as a one-time overhead to distill a concise prompt from demonstrations.
As shown in Figure 12, APE instructions reduce the number of prompt tokens by up to an order of magnitude compared to in-context learning.
Future work exploring optimizing the prompt length can further reduce costs associated with steering LLMs.
We the by multiplying and consumed by model type with OpenAI’s API rate as of September 1, 2022 (USD/1000 tokens: ada – 0.0004, babbage – 0.0005, curie – 0.0020, davinci – 0.0200).
Counter-intuitively, smaller models are more expensive.
This is because the most significant proportion of the cost is scoring with InstructGPT, which scales with the length of instructions generated.
Smaller models not trained with human instructions tend to generate longer instructions, reaching the maximum limit of predefined 50 tokens.
Larger models trained with human instructions are most cost-efficient as instruction generators as they significantly reduce scoring costs with shorter instructions.
E A DDITIONAL V ISUALIZATIONS
[Figure]
[Table] [Figure] [Figure]
x
[Figure]
c u t i on
A cc u r a cy [Figure] [Figure] u t i on
A cc u r a cy
[Figure] [Figure] [Figure] [Figure]
We compare the different models’ ability to propose instructions and use the same model (i
[Figure]
e.,
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
We compare the performance of different templates used to propose instruction.
Insert Template 1 is adapted from instruction induction
Template [Figure]
i n s t r u c t i on s w i t
h a cc u r a cy >
[Figure]
[Figure]
t r u c t i on s w i t
h a cc u r a cy >
[Figure] [Figure]
Starting
[Figure]
Pluralization
Passivization
Sentiment
Sentence
[Figure]
Word in
[Figure]
Negation
Antonyms
Synonyms
Membership
Rhymes
Large Animal
Cause Selection
[Figure]
Formality
Sum
Diff
Number to Word
Translation en-de Translation en-es
Translation en-fr1 E x e c u t i on
A cc u r a cy
GPT-3_S
GPT-3_M GPT-3_L
GPT-3_XL
InstructGPT_S
InstructGPT_M InstructGPT_L
InstructGPT_XL (a)
[Figure]
u t i on
A c c u r a cy
GPT
[Figure]
S
GPT
[Figure]
M
GPT-
[Figure]
_L
GPT-
[Figure] [Figure]
InstructGPT_S
InstructGPT
[Figure]
M
InstructGPT
[Figure]
InstructGPT_
[Figure]
(a)
[Figure]
t i on
A cc u r a cy
[Figure]
[Figure] [Figure]
cy
[Figure] [Figure]
t i on
A
[Figure]
u r a cy
[Figure]
[Figure] [Figure]
t
i
[Figure]
u r
[Figure]
cy
[Figure] [Figure]
% i n s t r u c t i on s w i t
h a cc u r a cy >
[Figure]
[Figure]
i n s t r u c t i on s w i t
h a cc u r a cy >
[Figure]
[Figure]
% i n s t r u c t i on s w i t
h a cc u r a cy >
[Figure]
oun
[Figure] [Figure]
i
[Figure]
s t r u c t
i on s w i t
h a cc u r a cy >
[Figure]
oun
[Figure]
[Figure]
% i n s t r u c t i on s w i t
h a cc u r a cy >
[Figure]
Train accuracy ( )
[Figure] [Figure]
C oun
[Figure] [Figure]
i n s t r u c t i on s w i t
h a cc u r a cy >
[Figure]
C
[Figure]