Prompt Engineering a Prompt Engineer (2024)

Abstract: Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models on customized tasks. It requires complex reasoning to examine the model’s errors, hypothesize what is missing or misleading in the current prompt, and communicate the task with clarity. While recent works indicate that large language models can be meta-prompted to perform automatic prompt engineering, we argue that their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. We fill this gap by infusing into the meta-prompt three key components: detailed descriptions, context specification, and a step-by-step reasoning template. The resulting method, named PE2, showcases remarkable versatility across diverse language tasks. It finds prompts that outperform “let’s think step by step” by 6.3% on MultiArith and 3.1% on GSM8K, and outperforms competitive baselines on counterfactual tasks by 6.9%. Further, we show that PE2 can make targeted prompt edits, rectify erroneous prompts, and induce multi-step plans for complex tasks.

## introduction large language models (llms) are powerful tools for many natural language processing tasks when given the right prompts. 1 however, due to the model's sensitivity @xcite @xcite , finding the optimal prompt can be challenging, often necessitating extensive manual trial-anderror efforts. moreover, once an initial prompt is deployed into production, unforeseeable edge cases may emerge, demanding more rounds of manual efforts to further refine the prompt. these challenges give rise to an emerging research field of † work done while interning at microsoft. 1 in this paper, we limit the scope to textual prompts of task description (e.g., "translate english to french") or instruction (e.g., "let's think step by step"). the prompt should be edited to guide the model to perform subtraction. propose a new prompt. let's solve this problem step by step. remember to add or subtract as needed. figure 1 : llm-powered automatic prompt engineering methods typically use a meta-prompt that guides an llm to inspect the current prompt , provide feedback (sometimes refered to as textual "gradients") and then generate an updated prompt . in this paper, we design and investigate meta-prompt variants to guide llms to perform automatic prompt engineering more effectively. automatic prompt engineering. within this field, a notable line of methods involves leveraging the capabilities of llms themselves @xcite . this entails meta-prompting llms with instructions such as "inspect the current prompt and a batch of examples, provide feedback, then propose a new prompt." (see figure 1 ) while these methods achieve impressive performance, a subsequent question arises: what makes a good meta-prompt for automatic prompt engineering? to answer this question, we connect two key observations: (1) prompt engineering, at its core, is a natural language generation task that requires complex reasoning: it involves closely examining the model's errors, hypothesizing what is missing or misleading in the current prompt, and communicating the task more clearly to the llm. the resulting method, named pe2, achieves strong empirical performance ( §5.1). when using text-davinci-003 as the task model, the prompts produced by pe2 surpass the zero-shot chain-ofthought prompt "let's think step by step" @xcite ) by 6.3% on multiarith and 3.1% on gsm8k. moreover, pe2 matches or outperforms two automatic prompt engineering baselines, iterative ape @xcite and @xcite in multiple settings (fig. 2 ). pe2 is most effective on counterfactual tasks @xcite , where the automatic prompt engineer is anticipated to reason about non-standard situations (e.g., do addition in base-8 instead of base-10) and explain such situation to the task model through the prompt. beyond academic datasets, we show that pe2 can improve an expert-written production prompt consisting of over 5,000 tokens, resulting in an 8.0% increase in f1 score. upon examining the prompt edit history of pe2 ( §5.3), we find that pe2 consistently offers meaningful prompt edits ( ## background in this section, we provide a formal formulation of the prompt engineering problem ( §2.1), and describe a general framework of automatic prompt engineering using llms and meta-prompts ( §2.2). building on this foundation, we introduce new meta-prompt components used in pe2 in §3. ## prompt engineering a prompt engineer much like how the prompt plays an important role for the end task performance, the meta-prompt p meta introduced in eq. 2 plays an important role in the quality of newly proposed prompts, and the overall quality of automatic prompt engineering. in this work, we focus on prompt engineering the meta-prompt p meta -we develop metaprompt components that can potentially help improve llms' prompt engineering quality. in the following, we reflect on the limitations of prior works and subsequently introduce three metaprompt components targeting these limitations. we visualize these components in fig. 3 . we name our method using these three components as pe2, a prompt engineered prompt engineer. ## conclusion in this paper, we introduced three meta-prompt components that lead to improved performance on automatic prompt engineering. the resulting method pe2 refines prompts written by human experts and surpasses established automatic prompt engineering baselines across various scenarios, notably on counterfactual tasks and production applications. through comprehensive analysis and case studies, we illustrate pe2's ability to make targeted prompt edits and generate high-quality prompts. the challenge of prompt engineering a prompt engineer remains ongoing. as highlighted in our case study, we believe improving the llm's instruction following abilities and mitigating hallucination issues will be crucial for improving automatic prompt engineering. as the capabilities of llm continue to evolve, their potential involvement in optimization or feedback loops necessitates a deeper empirical understanding of their failure modes, including shortcut learning discovered in this study. looking ahead, we are also excited about applying pe2 to optimize its own meta-prompt in a self-referential way, in line with metz et al. ( 2020) limitations firstly, we opt for a relatively small prompt search budget (@xmath0, @xmath1, @xmath2; see §4.3) due to cost considerations. in most of our experiments, the performance tends to plateau after @xmath3. however, it's important to consider that the use of the initialization prompt "let's think step by step" in many cases might introduce a potential confounding factor. this prompt could be already near-optimal, leading to fast convergence during prompt optimization. given the stochastic nature of natural language generation sampling and prompt optimization dynamics, it is possible that a larger prompt search budget or different experimental settings could yield new insights and conclusions. secondly, our study uses proprietary models such as gpt-4 and text-davinci-003. (1) it raises reproducibility concerns as proprietary models may undergo upgrades or discontinuation over time. however, we believe the core concepts introduced in this paper is model-agnostic. this is supported by our experiments where we use two different sets of (m proposal , m task ) (see §4.3). consequently, we anticipate that pe2 could be adapted for use with open models once their capabilities catch on. (2) it also raises concerns on data contamination, as the tasks and prompts included in our study may or may not have been part of the model's training data. lastly, apart from the three translation tasks in the instruction induction benchmark, our study predominantly focuses on tasks in english. we recognize the importance of inclusivity in language technology and acknowledge the need to extend our research to a multilingual setting in the future. seungone kim, se joo, doyoung kim, joel jang, seonghyeon ye, jamin shin, and minjoon seo. 2023b. the cot collection: improving zero-shot and few-shot learning of language models via chainof-thought fine-tuning. in proceedings of the 2023 conference on empirical methods in natural language processing, pages 12685-12708, singapore. association for computational linguistics. takeshi kojima, shixiang shane gu, machel reid, yutaka matsuo, and yusuke iwasawa. 2022. large language models are zero-shot reasoners. in advances in neural information processing systems. woosuk kwon, zhuohan li, siyuan zhuang, ying sheng, lianmin zheng, cody hao yu, joseph e. gonzalez, hao zhang, and ion stoica. 2023. efficient memory management for large language model serving with pagedattention. in proceedings of the acm sigops 29th symposium on operating systems principles. moxin li, wenjie wang, fuli feng, yixin cao, jizhi zhang, and tat-seng chua. 2023. robust prompt optimization for large language models against distribution shifts. in proceedings of the 2023 conference on empirical methods in natural language processing, pages 1539-1554, singapore. association for computational linguistics. yao lu, max bartolo, alastair moore, sebastian riedel, and pontus stenetorp. 2022. fantastically ordered prompts and where to find them: overcoming fewshot prompt order sensitivity. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 8086-8098, dublin, ireland. association for computational linguistics. aman madaan, niket tandon, prakhar gupta, skyler hallinan, luyu gao, sarah wiegreffe, uri alon, nouha dziri, shrimai prabhumoye, yiming yang, et al. 2023. self-refine: iterative refinement with self-feedback. arxiv preprint arxiv:2303.17651. luke metz, niru maheswaranathan, c daniel freeman, ben poole, and jascha sohl-dickstein. 2020. tasks, stability, architecture, and compute: training more effective learned optimizers, and using them to train themselves. arxiv preprint arxiv:2009.11243. maxwell nye, anders johan andreassen, guy gur-ari, henryk michalewski, jacob austin, david bieber, david dohan, aitor lewkowycz, maarten bosma, david luan, et al. 2021. show your work: scratchpads for intermediate computation with language models. arxiv preprint arxiv:2112.00114. openai. 2023. gpt-4 technical report. arxiv, abs/2303.08774. long ouyang, jeffrey wu, xu jiang, diogo almeida, carroll wainwright, pamela mishkin, chong zhang, sandhini agarwal, katarina slama, alex ray, et al. 2022. training language models to follow instructions with human feedback. advances in neural information processing systems, 35:27730-27744. debjit paul, mete ismayilzada, maxime peyrard, beatriz borges, antoine bosselut, robert west, and boi faltings. 2023. refiner: reasoning feedback on intermediate representations. arxiv preprint arxiv:2304.01904. archiki prasad, peter hase, xiang zhou, and mohit bansal. 2023. grips: gradient-free, edit-based instruction search for prompting large language models. in proceedings of the 17th conference of the european chapter of the association for computational linguistics, pages 3845-3864, dubrovnik, croatia. association for computational linguistics. reid pryzant, dan iter, jerry li, yin lee, chenguang zhu, and michael zeng. 2023. automatic prompt optimization with "gradient descent" and beam search. in proceedings of the 2023 conference on empirical methods in natural language processing, pages 7957-7968, singapore. association for computational linguistics. ning qian. 1999. on the momentum term in gradient descent learning algorithms. neural networks, 12(1):145-151. laria reynolds and kyle mcdonell. 2021. prompt programming for large language models: beyond the few-shot paradigm. in extended abstracts of the 2021 chi conference on human factors in computing systems, pages 1-7. chuck rosenberg, martial hebert, and henry schneiderman. 2005. semi-supervised self-training of object detection models. 2005 seventh ieee workshops on applications of computer vision (wacv/-motion'05) -volume 1, 1:29-36. subhro roy and dan roth. 2015. solving general arithmetic word problems. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 1743-1752, lisbon, portugal. association for computational linguistics. noah shinn, federico cassano, beck labash, ashwin gopinath, karthik narasimhan, and shunyu yao. 2023. reflexion: language agents with verbal reinforcement learning. arxiv preprint arxiv:2303.11366. jiao sun, yufei tian, wangchunshu zhou, nan xu, qian hu, rahul gupta, john wieting, nanyun peng, and xuezhe ma. 2023. evaluating large language models on controlled generation tasks. in proceedings of the 2023 conference on empirical methods in natural language processing, pages 3155-3168, singapore. association for computational linguistics. mirac suzgun, nathan scales, nathanael schärli, sebastian gehrmann, yi tay, hyung won chung, aakanksha chowdhery, quoc le, ed chi, denny zhou, and jason wei. 2023. challenging big-bench tasks and whether chain-of-thought can solve them. in findings of the association for computational linguistics: acl 2023, pages 13003-13051, toronto, canada. association for computational linguistics. 10 364