Automatic Engineering of Long Prompts (2024)

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in solving complex open-domain tasks, guided by comprehensive instructions and demonstrations provided in the form of prompts. However, these prompts can be lengthy, often comprising hundreds of lines and thousands of tokens, and their design often requires considerable human effort. Recent research has explored automatic prompt engineering for short prompts, typically consisting of one or a few sentences. However, the automatic design of long prompts remains a challenging problem due to its immense search space. In this paper, we propose an algorithm named Automated Prompt Engineering Xpert (APEX), a novel algorithm that automatically improves long prompts. Leveraging a greedy algorithm with beam-search for efficiency, APEX utilizes search history to significantly enhance the effectiveness of LLM-based mutation in its search process. Our results show that APEX achieves an average of 9.2% accuracy gain on eight tasks in Big Bench Hard and a consistent improvements on GSM8K with various models, highlighting the significance of automating prompt designs to fully harness the capabilities of LLMs.

## introduction large language models (llms) have exhibited remarkable capabilities when trained on large datasets, demonstrating the ability to comprehend complex and lengthy instructions for diverse tasks without the need for fine-tuning @xcite @xcite . several prompt design principles have emerged in recent years, suggesting that incorporating more complex instructions, demonstrations, and chain-of-thought reasoning into prompts can boost the performance on challenging tasks @xcite , including those involving mathematical problem-solving @xcite and reasoning @xcite . however, effective prompts for tackling complex tasks often contain thousands of tokens, posing challenges in designing and optimizing them. figure 1 demonstrates a long prompt for a task in big bench hard @xcite , which contains an instruction and several demos, each with humanwritten chain-of-thoughts. numerous studies have demonstrated the sensitivity of llms to prompts, revealing that minor modifications, such as adding or removing a few tokens or rephrasing the prompt, can significantly impact llm performance @xcite @xcite . therefore, prompt design has become a labor-intensive endeavor. further complicating the matter is the rapid evolution of llms, rendering prompts crafted for a specific llm ineffective when applied to a newer version of llm. this highlights the need for automatic prompt engineering techniques. in this paper, we consider the problem of automatic prompt engineering for black-box llms, where the tuning algorithms will improve prompts by querying the model. while automatic prompt engineering has been studied recently, existing research @xcite @xcite focuses on optimizing short instructions with one or a few sentences. these methods either evolve prompts by searching for word replacements or utilize llms to rewrite the entire prompt @xcite @xcite . it is challenging to apply them for evolving a long prompt like figure 1 . for long prompts, word replacement-based search faces a large search space, while rewriting the entire prompt using a single llm query is extremely difficult. although it is possible to apply these methods to individual sentences, we have observed that tuning a single sentence may not sufficiently improve the perfor-figure 1 : left panel: the average accuracy improvements of the proposed method over 3 runs with 50 iterations on the training set. we report the accuracy on the whole set (including training and test sets). more detailed results can be found in ## related work designing effective prompts to fully exploit the capabilities of llms has become an important topic. many principal ways for prompt design have been studied recently @xcite @xcite @xcite . for instance, a well-designed prompt may include a system prompt, an instruction prompt outlining the task, multiple contextual examples, and a chain-ofthoughts reasoning section that explains the stepby-step thought process behind the examples. by incorporating these elements, effective prompts often have tens of sentences and thousands of tokens. prompt engineering has been used to not only improving the performance of llms, but also to achieve certain goals, such as robustness testing @xcite and jailbreaking @xcite ). an orthogonal line of previous work has explored soft-prompt tuning, a technique that op-timizes prompts within a continuous embedding space using standard continuous optimization algorithms @xcite @xcite . while capable of achieving satisfactory performance, soft-prompts lack interpretability and cannot be applied via llm apis. additionally, these parameter-efficient fine-tuning methods demand large training sets, making them unsuitable for applications with limited data, such as those with only tens or hundreds of samples. given the limited availability of training data (e.g., < 1000 samples), our focus lies in exploring strategies for optimizing hard prompts, which are semantically equivalent to the original prompts but yield superior performance. the literature of automatic prompt engineering considers two primary settings. the first setting, which aligns with our work, assumes the existence of an initial humancrafted prompt and aims to refine or improve it to achieve enhanced performance. several discrete search algorithms have been proposed for this setting: @xcite searches for the trigger tokens that can improve the performance of language models on downstream tasks. @xcite employs a genetic algorithm for prompt tuning, utilizing back translation, cloze tasks, and sentence continuation to mutate the initial instruction. @xcite explored the utilization of input-output pair feedback to refine instructions in an iterative manner. more recently, @xcite proposed leveraging llms for mutation and crossover operations in evolutionary searches. although they can also be applied to entire prompt, we show in the main experiment that the proposed algorithm is more effective in evolving an entire prompt. this justifies the proposed ideas (history-guided evolution/sentence selection) can effectively isolate the important part of long prompt to evolve. in addition, @xcite demonstrated the optimization capabilities of llms in generating improved prompt variations based on previous fitness scores. however, these prompt evolution techniques are designed for short sentences or paragraphs within a long prompt. for example, many of them try to evolve only the instruction part or the sentence "let's think step by step" in the prompt. our work aims to provide complete freedom to evolve the entire long prompt, opening up more avenues for improvement but also introducing challenges in determining how and where to change the original prompt. other recent concurrent works have also used the idea of llm as agents to optimize prompt. @xcite proposed to model prompt optimization as a state-transition process, where at each state the llm uses the predicted error to generate feedback to and select next action. this work has a different search space with us, where they aim to correct existing prompts instead of rewriting it. another work that published after this paper focuses on incorporating human feedback into prompt optimization @xcite . another setting focuses on automatic prompt generation without a pre-existing prompt. @xcite demonstrated the ability of llms to generate brief task descriptions based on input-output pairs. building upon this technique, @xcite proposed an automatic prompt engineering (ape) algorithm capable of generating prompts from given pairs. under the same problem setting, @xcite developed a continuous relaxation approach. beyond these settings, there exists some algorithms developed to automate the generation of chain-of-thought prompts @xcite . our work seeks to design a general prompt tuner, which is orthogonal and can be combined with those instruction learning or auto chain-of-thoughts methods. ## proposed method in this paper, we address the challenge of automatic long prompt engineering for language models. given a language model l and an initial prompt p init designed by human for a particular task, our goal is to refine the prompt to achieve superior performance. to enhance the prompt, we are provided with a limited training set (e.g., 100 input-output pairs) {(x i , y i )} n @xmath0. specifically, for each sample we conduct prediction by l([p, x i ]) and assess its agreement with the corresponding ground truth label y i . we define score(p) as the performance metric of prompt p on the training set. we consider the scenario where the number of available samples n is limited, which is a common situation where individuals lack sufficient data for model fine-tuning but can still utilize the data to design an improved hard prompt. we will evaluate generalization performance on a hold-out test set that is not used in prompt search. ## conclusions we study the problem of automatic prompt engineering for long prompts, often comprising thousands of tokens. we investigate the performance of the standard greedy algorithm and genetic algorithm, and develop a search algorithm that yields superior performance. with only 50 evaluations on the training set, our method achieves an average absolute accuracy improvement of 9.2% across 8 tasks from big bench hard. this demonstrates the significant potential benefits of automatic long prompt tuning and underscores the importance of this emerging area.