PACE : Improving Prompt with Actor-Critic Editing for Large Language Model (2024)

Abstract: Large language models (LLMs) have showcased remarkable potential across various tasks by conditioning on prompts. However, the quality of different human-written prompts leads to substantial discrepancies in LLMs’ performance, and improving prompts usually necessitates considerable human effort and expertise. To this end, this paper proposes Prompt with Actor-Critic Editing (PACE) for LLMs to enable automatic prompt editing. Drawing inspiration from the actor-critic algorithm in reinforcement learning, PACE leverages LLMs as the dual roles of actors and critics, conceptualizing prompt as a type of policy. PACE refines prompt, taking into account the feedback from both actors performing prompt and critics criticizing response. This process helps LLMs better align prompt to a specific task, thanks to real responses and thinking from LLMs.We conduct extensive experiments on 24 instruction induction tasks and 21 big-bench tasks. Experimental results indicate that PACE elevates the relative performance of medium/low-quality human-written prompts by up to 98%, which has comparable performance to high-quality human-written prompts. Moreover, PACE also exhibits notable efficacy for prompt generation.

## introduction the rapid development of llms has led to notable advancements in artificial intelligence. llms, such as chatgpt @xcite , have emerged as essential tools in various application scenarios, including automatic question-answering @xcite , embodied agent @xcite , and code generation @xcite @xcite , among others. they have demonstrated remarkable capabilities in handling a range of tasks. however, their efficacy is not universal and often depends on how we interact with them -namely, how we provide appropriate prompts. @xcite , where each task contains about eight human-written prompts, with absolute performance differences between 29% and 93% for each task (refer to appendix c for detailed results). the interaction between users and llms is heavily mediated by prompts. these prompts can be understood as entry points, shaping and directing the llms' enormous reserves of knowledge and computational abilities toward specific outcomes. therefore, just as a key unlocks a door or a command instructs a computer program, prompts guide the response mechanisms of llms, determining the range and depth of answers they provide. figure 1 vividly illustrates the sensitivity of llms to the quality and specificity of prompts. even when posed with the same underlying task, varying the phrasing or approach of a prompt can yield vastly different results. for instance, while one prompt might retrieve a broad overview, another, only slightly rephrased (adding "in detail"), could elicit a detailed response. this variability underscores not only the importance of crafting thoughtful and effective prompts but also the nuanced complexities embedded within this task. the variability in llm performance with prompts is primarily caused by two key factors: 1. human articulation limitations: humans inherently think and communicate in a manner that is filled with nuances, emotions, and subjectivities. when posing questions or presenting requirements, they might inadvertently omit vital details or introduce ambiguities. our natural way of communication is also peppered with cultural references, idioms, and shorthand that might not always be clear or universally understood. therefore, humanwritten prompts can sometimes express incomplete, ambiguous, or even erroneous requirements. this not only affects the accuracy of the output but might also skew it in unintended directions. 2. cognitive discordance between humans and llms: even when we assume that a prompt is perfectly articulated, another challenge arises. there is an intrinsic cognitive gap between human comprehension and the way language models interpret information. human comprehension and cognition need to be "translated" into expressions that align with llm's expectations. thus, a well-phrased requirement from a human perspective may still lead to an llm output that feels off or unexpected to humans. as a result, crafting high-quality prompts is usually not accomplished in one go but requires trial and error. in general, humans develop prompts in two stages: 1. summarizing the initial prompt: the first stage involves crafting a preliminary version of a prompt. it is a process that often draws upon existing data, prior knowledge, or a specific need within a given context. 2. improving and editing prompts: the second stage is characterized by a continuous process of improvement, modification, and fine-tuning. this stage is crucial because it takes the initial draft to a polished level, where the prompt becomes more precise, clear, and potentially more effective in eliciting the desired response from an llm. this stage often involves iterative feedback loops, close scrutiny. however, existing approaches of automatic prompt engineering concentrate primarily on the first stage @xcite @xcite , while overlooking the second stage that significantly enhances the quality of prompts. intuitively, the effect and human effort of the second stage far surpass those of the first stage, as it is relatively straightforward for humans to draft a preliminary version of the prompt. consequently, there emerges an imperative need for advancements in automatic prompt editing for llms. in this paper, we propose an effective approach named pace to tackle automatic prompt editing for llms. this approach draws inspiration from the actor-critic algorithm, a well-known technique in the realm of reinforcement learning. pace repurposes llms as both the actor and the critic. conceptually, a prompt can be viewed as a policy that directs the behavior of the llm. the actor, or the llm, performs tasks based on this policy (prompt), while the critic provides a form of supervision, identifying how well the actor is performing based on the provided prompts. on this basis, pace improves the quality of the prompt, thereby optimizing the performance of llms on specific tasks. we conduct extensive experiments to evaluate the effectiveness of pace on 24 instruction induction tasks and 21 big-bench tasks. the experimental results indicate the effectiveness of pace for automatic prompt editing and generation. ## pace we consider a task t , accompanied by demonstration data @xmath0, y )}. for the task t , given an input x, a corresponding desired output is y . the objective of automatic prompt editing is to identify a prompt p such that, when a llm m is queried with the concatenation of this prompt and a specified input [p; x], it generates the corresponding output y . therefore, we aim to find the prompt p that maximizes the expectation of score s(p, x, y ) over possible pairs (x, y ). owing to the vast and potentially infinite search space, there are significant challenges in obtaining optimal prompts. following the previous work @xcite @xcite , we leverage the capabilities of llm to approximate the inference of the most likely prompt p with a high score. to further augment the proficiency of llm in generating or refining prompts, we introduce the pace approach, which consists of the actor-critic paradigm and iterative algorithm. ## experimental results in this section, we detail the results of our comprehensive experiments which offer compelling evidence of the effectiveness of pace in improving the performance of llms. note that in most experiments, we only present the average results from all experiments for each benchmark. detailed results for each task are available in the appendix. ## related work recent advancements in transformer-based llms have not only improved the model's performance across various nlp tasks @xcite @xcite but have also revealed emergent capabilities, including fewshot in-context learning, zero-shot problem solving, chain of thought reasoning, instruction following, and instruction induction @xcite @xcite @xcite @xcite . while we share the sentiment with these works about the potential of llms, our focus lies in enhancing their performance through prompt editing strategies. ## conclusion and discussion in this paper, we have proposed pace, an innovative approach to automatic prompt editing for llms, drawing inspiration from the actor-critic paradigm in reinforcement learning. our experiments confirm the potential of pace in significantly enhancing the effectiveness of prompts, leading to improved llm performance across a variety of tasks. by treating the prompt as a form of policy and conceptualizing llms as both actors and critics, we have presented a fresh perspective on how prompts can be optimized to better guide the output of llms. the remarkable improvements with pace in prompt editing and generation underscore the value of this perspective and its potential to transform the field of prompt engineering.