PR ewrite: Prompt Rewriting with Reinforcement Learning (2024)

Abstract: Prompt engineering is critical for the development of LLM-based applications. However, it is usually done manually in a “trial and error” fashion that can be time consuming, ineffective, and sub-optimal. Even for the prompts which seemingly work well, there is always a lingering question: can the prompts be made better with further modifications?To address these problems, we investigate automated prompt engineering in this paper. Specifically, we propose PRewrite, an automated method to rewrite an under-optimized prompt to a more effective prompt. We instantiate the prompt rewriter using an LLM. The rewriter LLM is trained using reinforcement learning to optimize the performance on a given downstream task. We conduct experiments on diverse benchmark datasets, which demonstrates the effectiveness of PRewrite.

## introduction with the right prompts, large language models (llms) can show impressive performance on various tasks in zero-shot or few-shot settings @xcite . however, manual prompt engineering is done on a trial-and-error ad-hoc basis and there are limited guiding principles on writing good prompts. to address the problems, we investigate methods to automate the process of prompt engineering, often called "automated prompt engineering" or "prompt optimization". automated prompt engineering is important due to the wide and fast adoption of llm applications. moreover, llms themselves are evolving, and as a result, we also need effective automated methods to update existing prompts to adapt to new models. several previous works have explored automated prompt engineering. autoprompt @xcite uses a gradient-based search method to iteratively edit prompts, but requires gradient access to the language model. rlprompt @xcite optimizes prompts using reinforcement learning (rl), but often produces uninterpretable gibberish prompts. also using rl, tempera @xcite allows editing prompts based on task input, but its small action space might hinder exploration. another common limitation is that they are based on relatively small-size language models like bert @xcite and roberta @xcite . it is not clear how well the proposed methods can generalize to larger models, especially with api-only model access. more recent works like ape @xcite , opro @xcite and promptbreeder @xcite use larger models from the palm 2 @xcite and the gpt foot_0 model families. these works leverage llms themselves to propose prompt candidates, and search for a better prompt from them via validating performance on a given training dataset. we follow a similar idea but aim to use rl instead of search to improve the optimization process. in this work, we propose prewrite, prompt rewriting with reinforcement learning, to address the limitations above. our idea is to train a prompt rewriter to rewrite an initial under-optimized prompt to a more effective prompt. the prompt rewriter itself is a llm, trained using rl to optimize for a downstream task. we give an overview in figure 1 . specifically, given an initial prompt, the prompt rewriter llm is instructed to generate a rewritten prompt, which in turn is used by the task llm to generate the final output. using a reward computed on the final output against the ground-truth output, the rewriter llm is finetuned with rl. as compared to previous rl-based methods, prewrite produces interpretable prompts (cf. rl-prompt), allows unconstrained exploration without manually defined action space (cf. tempera), and leverages larger models (palm 2). our contributions are summarized as follows: • we propose prewrite, a novel automated prompt engineering approach. it optimizes prompt via rewriting, in an end-to-end manner using reinforcement learning. ## related work we survey related work on automated prompt engineering for discrete prompts. please refer liu et al. nq: "answer the question" → "compose a short, informative answer that directly answers the given question. the answer should be no longer than 15 words and should not contain any extraneous information. for example, if the question is "who is the president of the united states?", the answer should be "joe biden". do not write an essay or provide additional explanation." gsm8k: "solution"" → "solve the problem by following the steps in the solution." some earlier works optimize prompts via paraphrasing @xcite @xcite . in contrast, we adopt a powerful llm to rewrite prompts, providing more capacity for prompt optimization. @xcite ; @xcite propose gradient-based search approach which is challenging for larger api-access only models as it requires model gradient access. prior work have also explored rl based solutions. rlprompt @xcite optimizes prompts using rl, but often produces uninterpretable gibberish prompts. tempera @xcite allows prompt editing at test time based on task input using rl, but it defines a small action space. by leveraging more capable llms, our method produces interpretable prompts and allows unconstrained exploration without a manually defined the action space. more recent works use blackbox llms such as palm 2 and gpt models, similar to ours. these include ape @xcite , promptbreeder @xcite and opro @xcite . these works use llms in different ways to propose prompt candidates and search for the optimal one via validating performance on a given training dataset. we follow a similar idea but instead use rl for prompt optimization. ## conclusions in this paper, we present prewrite, a prompt rewriter trained with reinforcement learning (rl) for prompt optimization. we instantiate the rewriter with a llm (palm 2-s) and finetune it using rl to optimize the end task performance. to further improve the performance, we develop a rewriting strategy that searches from the rewritten prompts generated by the trained rewriter. our experiments testify the effectiveness of prewrite and demonstrate its state-of-the-art performance.