Black-Box Prompt Optimization: Aligning Large Language Models without Model Training (2024)

Abstract: Large language models (LLMs) have shown impressive success in various applications. However, these models are often not well aligned with human intents, which calls for additional treatments on them; that is, the alignment problem. To make LLMs better follow user instructions, existing alignment methods primarily focus on further training them. However, the extra training of LLMs is usually expensive in terms of GPU computing; even worse, some LLMs are not accessible for user-demanded training, such as GPTs. In this work, we take a different perspective—Black-Box Prompt Optimization (BPO)—to perform alignments. The idea is to optimize user prompts to suit LLMs’ input understanding, so as to best realize users’ intents without updating LLMs’ parameters. BPO leverages human preferences to optimize prompts, thus making it superior to LLM (e.g., ChatGPT) as a prompt engineer. Moreover, BPO is model-agnostic, and the empirical results demonstrate that the BPO-aligned ChatGPT yields a 22% increase in the win rate against its original version and 10% for GPT-4. Notably, the BPO-aligned LLMs can outperform the same models aligned by PPO and DPO, and it also brings additional performance gains when combining BPO with PPO or DPO. Code and datasets are released at https://github.com/thu-coai/BPO.

## introduction recently, the field of natural language processing has made remarkable progress, largely thanks to the advent of large language models (llms) @xcite @xcite @xcite . after elaborate alignment @xcite , these models have demonstrated a strong ability of instruction-following and human preference understanding, yielding products like chatgpt (openai, 2022) that have attracted widespread attention. however, aligning llms to human preferences is not trivial. the major challenge lies in narrowing the gap between human intents (conveyed by prompts) and llms' understanding of them. significant effort has been focused on steering llms to approach human preference, including reinforcement learning from human feedback (rlhf) @xcite , reinforcement learning from ai feedback (rlaif) @xcite , or direct preference optimization (dpo) @xcite . nevertheless, these methods suffer from various deficiencies: distinct from the aforementioned alignment methods, we propose to steer human prompts to accommodate llms' understanding. while the idea is closely related to "prompt engineering", its automated prototypes would trace back to autoprompt @xcite and prompt tuning (i.e., p-tuning) @xcite , where prompts are optimized to improve task performance without training the lms. our new alignment method, black-box prompt optimization (bpo), presents an efficient and interpretable paradigm that aligns llms without modifying these models. the central idea behind bpo is to create an automatic prompt optimizer that rewrites human prompts, which are usually less organized or ambiguous, to prompts that better deliver human intent. consequently, these prompts could be more llm-preferred and yield better human-preferred responses. in bpo, the prompt preference optimizer is learned from preference comparisons. we curate a subset of publicly available sft datasets with either human or ai preferences. each instance of our training data contains a prompt along with a pair of favorable and unfavorable responses. we then employ llms to delineate and criticize the paired responses, and subsequently ask the llms to refine the input prompt to explicitly incorporate the features that shift the responses from unfavorable to favorable. in this way, we construct 14k pairs of the original instruction and its optimized version to train a sequence-to-sequence model that optimizes user instructions. our extensive experiments demonstrate that without llm training, bpo can improve the alignment of both api-based and open-sourced llms remarkably: increasing win rates by 8.8% to 22.0% on gpt-3.5-turbo, gpt-4, claude-2, llama-2-chat, vicuna etc. moreover, we show that bpo not only outperforms rlhf via ppo @xcite and dpo @xcite but also further improves llms' alignment after these rlhf's training. we also show that bpo can align llms in supervised fine-tuning by optimizing response quality in the experiment of alpaca. in addition, we have demonstrated the superiority of bpo over the direct use of llm as a prompt engineer, highlighting the importance of incorporating human feedback. our contributions can be summarized as follows: ## related work llms pre-trained on massive corpus can generate fluent text but are not well aligned to follow users' instructions. therefore, aligning llms with human intents has become an important research problem. existing efforts in alignment mostly follow the paradigm proposed by @xcite , consisting of two main stages: sft and rlhf. supervised fine-tuning (sft). sft alignment endows llms with preliminary instruction-following abilities. nonetheless, it heavily relies on abundant high-quality fine-tuning data. since the high cost of human-written data, self-instruct data augmentation @xcite based on a small human-created seed set has become a predominant approach in academia @xcite . however, sft alignment still suffers from hallucinations, inferior scalability, and poor understanding of human preference. ## black-box prompt optimization the overall process of bpo is shown in figure 2 . bpo is to enhance the alignment between model output and human preference by optimizing the input prompt. to this end, we first collect several ## interpretability of bpo compared with model-training-based alignment methods like ppo or dpo, bpo has a distinct advantage in its strong interpretability, as we can directly compare the instructions before and after optimization to find out how bpo works. to examine what bpo optimizes in detail, we closely examined 500 samples and summarized some common patterns in its optimization and error types. as shown in figure 4 , we summarize four common optimization strategies exhibited in bpo's results, including explanation generation (green box), prompt elaboration (orange box), providing hint (blue box) and safety enhancement (pink box). we should note that there are also other optimization strategies observed in bpo's output, and those strategies are not mutually exclusive. these presented examples are only typical instances in these four categories. when user inputs could potentially raise security issues, bpo emphasizes maintaining harmless responses. moreover, bpo enables interpretable security enhancements, as it can refine the unsafe request to require the model to output relevant harmless advice. in this way, we can better prevent safety issues while still keeping responses helpful. error analysis is shown in appendix i. ## conclusion in this work, we present bpo, a black-box alignment method that automatically optimizes user inputs to better suit llms' preference for improved responses. with bpo alignment, we successfully improve the alignment of llms without further adjusting these models, leading to significant results even on the most powerful models like gpt-4 and claude-2. moreover, extensive experiments show that bpo can reach or surpass the performance of current mainstream alignment techniques on vicuna models and further improve these alignment methods. our findings demonstrate that tailoring inputs to best suit llms is a promising technical direction to obtain interpretable and controllable alignment in parallel to existing model-trainingbased solutions, and there is still great room to further explore in depth.