PR ompt Optimization in Multi-Step Tasks ( PROMST ): Integrating Human Feedback and Heuristic-based Sampling (2024)

Abstract: Prompt optimization aims to find the best prompt to a large language model (LLM) for a given task. LLMs have been successfully used to help find and improve prompt candidates for single-step tasks. However, realistic tasks for agents are multi-step and introduce new challenges: (1) Prompt content is likely to be more extensive and complex, making it more difficult for LLMs to analyze errors, (2) the impact of an individual step is difficult to evaluate, and (3) different people may have varied preferences about task execution. While humans struggle to optimize prompts, they are good at providing feedback about LLM outputs; we therefore introduce a new LLM-driven discrete prompt optimization framework PROMST that incorporates human-designed feedback rules to automatically offer direct suggestions for improvement. We also use an extra learned heuristic model that predicts prompt performance to efficiently sample from prompt candidates. This approach significantly outperforms both human-engineered prompts and several other prompt optimization methods across 11 representative multi-step tasks (an average 10.6%-29.3% improvement to current best methods on five LLMs respectively). We believe our work can serve as a benchmark for automatic prompt optimization for LLM-driven multi-step tasks.

## introduction the performance of large language models (llms) on a given task is sensitive to the prompt, so prompt engineering aims to create prompts that fully leverage the capabilities of llms. due to the lack of access to model parameters for black-box llms, techniques for automatic prompt optimization have primarily focused on searching over the vast discrete space of tokenized language inputs @xcite as it interacts with an environment over multiple steps @xcite . engineering good prompts is hard due to the typical prompt length (300+ tokens) and individual task constraints and rules. the prompts needed for multi-step tasks are more complex to judge the long-horizon correctness of a single action. this difficulty hinders llms from automatically reasoning over errors and producing better prompts, which in turn reduces the effectiveness of current methods for automated prompt optimization. prompt optimization in multi-step tasks is still an open challenge. considering that humans excel in analyzing errors and incorporating relevant domain knowledge into feedback, we formalize promst as a framework involving human input, as shown in figure 1 . here, during the multi-step agent-environment interactions, the agent (indicated by 'taskllm' in figure 1 ) sometimes makes errors and fails the task. figure 1 : the promst framework. given an initial human-designed prompt and the state of the environment for the current task, the taskllm iteratively generates an action and executes it until either an error occurs or the task is complete. human-designed feedback rules automatically generate feedback about errors that is then provided as context to the promptllm when generating new prompt candidates. the task performance is scored according to a human-designed score function; this score can be used with the prompt to train a score prediction model online. given new prompt candidates, this score prediction model is used to select a subset of candidates to evaluate for the next generation. while some work has used llms to evaluate errors, we instead use human-designed feedback rules constructed a priori that address different types of errors. depending on the error, feedback is automatically generated and passed as additional context to an llm that is responsible for producing a new set of candidate prompts (indicated by 'promptllm' in figure 1 ). a score is assigned to each prompt indicating the agent's task performance given that prompt. since the evaluation of many candidate prompts for multi-step tasks in environments can be expensive, we fine-tune a score prediction model online using prompt-score pairs which can be used as a heuristic to select a subset of the candidate prompts to evaluate. our experiments in 11 tasks show that the integration of human feedback and the score model greatly improves the prompt optimization process (10.6%-29.3% relative improvements over all baseline methods across different llms). promst achieves the best performance on most tasks. promst has also been shown to perform better in multi-trial settings when combined with dynamic approaches. we further show that the humandesigned evaluation rules can be used to help align task performance with human preferences. extensive experiments are conducted to validate the framework and investigate the underlying reasons why some prompts are more effective than others. in summary, our contributions are : (1) to our best knowledge, promst is the first to explore automatic prompt optimization in multi-step agent tasks. we release all codes and prompts for 11 multi-step environments, which may serve as a benchmark for future research. (2) we show that the integration of human feedback and a fine-tuned score model outperforms existing methods across various tasks and llms. (3) our research indicates that promst is orthogonal and integrates well with established dynamic approaches. (4) we find that human-designed rules for task evaluation help align optimized prompts with human preferences. ## related work prompt optimization to improve performance of black-box api models, it is useful to engineer the discrete prompts for downstream tasks. various 'best practices' have emerged for human-designed task prompts, such as including examples (brown et al., our work focuses on domains that include complex multi-step tasks, in which the evaluation and reflection processes are more challenging so that score prediction models and human feedback rules are introduced in order to mitigate this problem. another type of methods is to utilize selfreflection for online dynamic feedback during task execution, such as reflexion @xcite . while reflexion needs multiple trials for online optimization of action memory for each single test, our method promst only needs one trial for task execution and the prompt is optimized offline across multiple tests. in section 4.4, we find that promst outperforms the offline variation of reflexion and can perform better when combining with online reflexion in the multi-trial setting. ## conclusion in this work we introduce an automatic prompt optimization framework for complex, multi-step agent tasks: promst. to handle the issues of task complexity, judging long-horizon correctness of individual actions, high prompt exploration cost,